Generating accurate augmented reality objects in relation to a real-world surface via a digital writing device

ABSTRACT

The present disclosure includes systems, methods, computer readable media, and devices that can generate accurate augmented reality objects based on tracking a writing device in relation to a real-world surface. In particular, the systems and methods described herein can detect an initial location of a writing device, and further track movement of the writing device on a real-world surface based on one or more sensory inputs. For example, disclosed systems and methods can generate an augmented reality object based on pressure detected at a tip of a writing device, based on orientation of the writing device, based on motion detector elements of the writing device (e.g., reflective materials, emitters, or object tracking shapes), and/or optical sensors. The systems and methods further render augmented reality objects within an augmented reality environment that appear on the real-world surface based on tracking the movement of the writing device.

BACKGROUND

Recent years have seen rapid development in systems and devices forgenerating drawings within a digital workspace. Indeed, developers havegenerated digital composition systems that can gather user input andgenerate digital representations to allow users to create digitaldrawings. For example, some digital composition systems utilize a devicewith a touchscreen to track user input and generate a drawing on thetouchscreen.

Although such conventional digital composition systems can createdrawings within a digital workspace, they have a number of significantshortcomings. For example, conventional digital composition systemsrequire a specific device and drawing surface, such as a touchscreen, onwhich a user must draw or write. Some of these conventional digitalcomposition systems also require specific devices to, for example,compose in different colors. Accordingly, these conventional compositionsystems require users to transport devices with drawing surfaces, alongwith multiple writing implements, in order to create digital drawings.These systems thus restrict where (e.g., on what surface) a user cancompose and further restrict the size of the composition space to thedimensions of the given drawing surface. Furthermore, drawing surfaces,such as touchscreens, are often difficult to utilize because of a lackof visibility caused by environmental factors, such as glare, darkness,or excessive light.

Some systems have sought to correct these shortcomings by utilizingaugmented reality technology. For example, conventional augmentedreality systems can track general movement of a hand, finger, or largehandheld controller to generate rough digital shapes. While theseconventional augmented reality systems can create rough digitalrepresentations, they are imprecise and inaccurate. In particular,conventional digital composition systems that utilize augmented realitytechnology are inexact and struggle to track and represent detailedmovements in a digital object. As a result of this inaccuracy,conventional digital composition systems prevent users from drawing orwriting with any appreciable aesthetic detail.

Thus, there are several disadvantages with regard to conventionaldigital composition systems.

SUMMARY

One or more embodiments described herein provide benefits and solve oneor more of the foregoing or other problems in the art with systems,methods, non-transitory computer-readable media, and devices forgenerating digital objects in an augmented reality environment bytracking a writing device in relation to a real-world surface. Inparticular, the systems described herein can detect (e.g., via a noveldigital writing device) a user drawing on an existing surface in areal-world environment (e.g., a table, desk, or wall), generate anaugmented reality object based on the detected drawing, and project theaugmented reality object onto the existing surface within an augmentedreality environment (e.g., such that it appears to the user that theyare drawing on the table, desk, or wall). The disclosed systems can thusprovide an accurate, sensitive tool for generating drawings that appearto be located on a variety of surfaces within a real-world environmentand without the need for a device that includes a separate drawingsurface for receiving, monitoring, and/or tracking user input.

For instance, in one or more embodiments, the disclosed systems includea writing device that utilizes sensors and other components toaccurately and precisely track interactions between a user, writingdevice, and writing surface, thereby enabling the system to renderaugmented reality objects (e.g., using an augmented reality device) at ahigh level of detail. To illustrate, a user may use the writing deviceto draw on a table or other surface, and, in response, the systemdescribed herein may detect the contact of the writing device with thetable (e.g., by way of a pressure sensor) and may track the motion ofthe writing device by way of a motion detector. The system may furtheranalyze the real-world movement of the writing device and generate adigital object based on the real-world movement. Further, the disclosedsystems can render the digital object by way of the augmented realitydevice such that the digital object appears on the table or othersurface.

The disclosed systems, methods, computer-readable media, and devicesprovide a number of improvements over conventional digital compositionsystems. As an initial matter, the disclosed systems can generateaugmented reality objects without the need for a digital writingsurface. Thus, in one or more embodiments, the disclosed systems canexpand available creative space (by removing boundaries inherent totouchscreens or other existing writing surfaces) while also freeingusers from the need to transport large devices that include built-inwriting surfaces. Furthermore, the disclosed systems can provideaccurate and highly detailed renderings of user compositions. Indeed,the disclosed system can track changes in pressure, tilt, orientation,location and movement of a writing device in relation to a real-worldsurface and generate augmented reality objects that reflect detailednuances of expressive strokes, shading, and styles. In addition, thedisclosed systems can generate augmented reality objects within anaugmented reality object that are easily visible, regardless ofsurrounding environmental factors (e.g., by dimming surroundingenvironmental distractions or emphasizing augmented reality objects).Furthermore, the disclosed systems can improve performance ofimplementing computing systems by selecting (and excluding) techniquesfor tracking a writing device in response to circumstances unique to aparticular real-world environment

Additional features and advantages of the present application will beset forth in the description which follows, and in part will be obviousfrom the description, or may be learned by the practice of such exampleembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

This disclosure will describe one or more embodiments of the inventionwith additional specificity and detail by referencing the accompanyingfigures. The following paragraphs briefly describe those figures, inwhich:

FIG. 1 illustrates an example schematic diagram of an exampleenvironment of an augmented reality composition system in accordancewith one or more embodiments;

FIG. 2 illustrates an example augmented reality object generated basedon tracking interaction with an example writing device on an examplereal-world surface in accordance with one or more embodiments;

FIG. 3A illustrates an example writing device including example sensorsin accordance with one or more embodiments;

FIG. 3B illustrates an example augmented reality device tracking anexample writing device in accordance with one or more embodiments;

FIG. 3C illustrates an example augmented reality device tracking anexample writing device in accordance with one or more embodiments;

FIG. 3D illustrates an example augmented reality device tracking anexample writing device in accordance with one or more embodiments;

FIG. 3E illustrates an example augmented reality device tracking anexample writing device in accordance with one or more embodiments;

FIG. 3F illustrates an example writing device on an example real-worldsurface in accordance with one or more embodiments;

FIG. 4 illustrates identifying and displaying real-world surfacesacceptable for composing via an example augmented reality environment inaccordance with one or more embodiments;

FIG. 5 illustrates an example table of approaches for tracking a writingdevice in accordance with one or more embodiments;

FIG. 6A illustrates an example writing device including one or morecomponents in accordance with one or more embodiments;

FIG. 6B illustrates an example writing device including one or morecomponents in accordance with one or more embodiments;

FIG. 7 illustrates an example schematic diagram of an augmented realitycomposition system in accordance with one or more embodiments;

FIG. 8 illustrates a flowchart of a series of acts in a method forrendering an augmented reality object in accordance with one or moreembodiments;

FIG. 9 illustrates a block diagram of an example computing device inaccordance with one or more embodiments.

DETAILED DESCRIPTION

One or more embodiments described herein provide benefits and solve oneor more of the foregoing or other problems in the art with an augmentedreality composition system that generates digital objects in anaugmented reality environment by accurately tracking a writing device inrelation to a real-world surface. In particular, the augmented realitycomposition system can accurately detect interaction between a writingdevice and a real-world surface that reflects drawing (or writing) onthe real-world surface. In response, the augmented reality compositionsystem can generate an augmented reality object in real-time thatcorresponds to the user input (e.g., the movement of the writing device)within a three-dimensional augmented reality environment. Moreover, theaugmented reality composition system can portray the augmented realityobject on the real-world surface (e.g., via an augmented reality device)such that it appears to the user that drawing lies on the real-worldsurface.

To illustrate, the augmented reality composition system can detect aninitial contact of a writing device with a real-world surface. Forexample, the augmented reality composition system can detect the initialcontact by way of a pressure sensor connected to a tip of the writingdevice. Additionally, the augmented reality composition system candetermine an initial location of the writing device in response todetecting the initial contact of the writing device with the real-worldsurface. For instance, the augmented reality composition system candetermine the initial location via a locator element (e.g., laseremitter or infrared reflective element) affixed to the writing device.Furthermore, the augmented reality composition system can detectmovement (e.g., real-world movement) of the writing device relative tothe initial location by way of a motion detector element (e.g., opticalsensor that detects light deflections) associated with the writingdevice. Based on the detected movement of the writing device, theaugmented reality composition system can render an augmented realityobject that, when observed by way of an augmented reality device,appears to be located on the real-world surface.

As just mentioned, the augmented reality composition system can generatedigital objects within an augmented reality environment. Moreparticularly, the augmented reality composition system can trackpressure, orientation, location, and movement of a writing device andrender augmented reality objects. To generate an augmented realityenvironment and augmented reality objects consistent with a real-worldenvironment, the augmented reality composition system can utilize athree-dimensional modeling algorithm. For example, in one or moreembodiments, the augmented reality composition system utilizes athree-dimensional modeling algorithm that combines depth informationtogether with color data (e.g., Red Green Blue (“RGB”) data) to generatea three-dimensional model of a real-world environment.

In addition, the augmented reality composition system can utilize anumber of techniques to track the initial location and movement of thewriting device (or more precisely movement of a tip of the writingdevice). For example, the augmented reality composition system canperform object recognition techniques to track the shape of the writingdevice as it moves in the real world. Moreover, the augmented realitycomposition system can track the writing device by identifying anindicator such as, for example, a laser emitted by the writing deviceand projected onto the real-world surface. Similarly, the augmentedreality composition system can track the writing device by detectingmovement and/or orientation of a material that reflects a particularwavelength of light (e.g., infrared tape). Furthermore, the augmentedreality composition system can track the writing device by way of amotion detector, such as an inertial measurement unit (“IMU”) on thewriting device, or by an optical sensor that detects light deflectionoff of real-world surfaces as the writing device.

Utilizing various sensors and components, the augmented realitycomposition system can generate augmented reality objects that reflectnuanced strokes, shading, drawing, or other marks. For example, thewriting device can include a pressure sensor that is configured todetect contact of the writing device on a surface. The pressure sensormay further be configured to detect variations in pressure applied tothe writing device (e.g., the tip of the writing device). In addition,the writing device can include a locator element that detects variationsin orientation (i.e., tilt) of the writing device. Based on variationsin pressure, tilt, and/or orientation, the augmented reality compositionsystem can vary digital marks. To illustrate, the augmented realitycomposition system can modify weight or thickness of the digital mark toallow for broad shading or more detailed, defined lines based onvariations in pressure and/or orientation.

The augmented reality composition system can also include variousfeatures allowing a user to modify, manipulate, or share augmentedreality objects. For instance, because the augmented reality compositionsystem generates an augmented reality environment based on athree-dimensional analysis of real-world surroundings, the augmentedreality composition system can further change three-dimensionalcoordinates of an augmented reality object (e.g., render a movinganimation of the drawing that follows the movement of the writing devicethrough the air). Similarly, the augmented reality composition systemcan render interactive digital objects, such as a digital color paletteor a digital menu, within an augmented reality environment and detectuser interaction with the digital objects via the writing device.Furthermore, the augmented reality composition system can transfer anaugmented reality object from one augmented reality device to another(e.g., by pushing, throwing, or otherwise moving the augmented realityobject in the direction of the other user).

The augmented reality composition system provides various benefits overconventional composition systems and conventional virtual reality (“VR”)composition systems. For example, the augmented reality compositionsystem is more precise and accurate than conventional VR compositionsystems. By using various sensitive tracking techniques, the augmentedreality composition system can accurately track interactions between awriting device and real-world surface. Indeed, the augmented realitycomposition system can detect even slight movement, tilt, pressure, andother sensory information related to the writing device. By analyzingthese various factors in isolation and/or in conjunction with oneanother, the augmented reality composition system can more accuratelyinterpret how a user moves, pushes on, lifts up, or otherwisemanipulates a writing device and generate more precise augmented realityobjects (e.g., that realistically mimic writing or drawing on areal-surface).

Furthermore, the augmented reality composition system provides increasedflexibility. In particular, where some conventional systems require auser to write on a specific medium (e.g., a touchscreen), in one or moreembodiments, the augmented reality composition system enables a user towrite or draw on virtually any surface. To illustrate, the augmentedreality composition system analyzes real-world surfaces and providesindications to the user of which surfaces are acceptable for composing,whereupon the user can freely write on an acceptable surface.Accordingly, users no longer need to purchase and transport devices withwriting surfaces and are not limited by the boundaries of such writingsurfaces.

Additionally, the augmented reality composition system enables a user tomove an augmented reality composition from one surface to another. Toillustrate, the augmented reality composition system can save anaugmented reality object that the user composes on a first real-worldsurface and, in response to detecting user input to load the savedaugmented reality object onto a different real-world surface (e.g., in adifferent room), the augmented reality composition system loads theaugmented reality object onto the new real-world surface. Thus, theaugmented reality composition system enables the user to edit andcontinue working on augmented reality compositions wherever the user islocated.

The augmented reality composition system can also improve visibilitywhile composing on a real-world surface. For example, in one or moreembodiments, the augmented reality composition system can furtherhighlight augmented reality objects composed by the user. To illustrate,the augmented reality composition system can increase the brightness ofan augmented reality object to help the augmented reality object standout against a background. Similarly, the augmented reality compositionsystem can dim a surrounding area to emphasize a user's workspace. Forexample, the augmented reality composition system can dim the areasoutside of the real-world surface on which the user is currentlycomposing. Alternatively, the augmented reality composition system canproject augmented reality objects (e.g., computer-generated digitalobjects or user-created digital objects such as digital photographs)such as a white sheet on a real-world surface to improve visibility.

As still another advantage of the augmented reality composition systemover conventional systems, the augmented reality composition systemgives users a more organic composition experience. Whereas someconventional systems are limited by digital screens and writingsurfaces, the augmented reality composition system described hereinmimics different textures and colors of real art supplies such as paint,pen, pencil, etc. For example, by utilizing an immersive,three-dimensional augmented reality environment, the augmented realitycomposition system provides more realistic approximations of actualcolors and textures of various art supplies.

Furthermore, the augmented reality composition system can utilizecomputing resources more efficiently than conventional compositionsystems. For example, in one or more embodiments, the augmented realitycomposition system analyzes real-world surroundings or circumstances andselects techniques based on the detected circumstances. By selectingparticular techniques (and omitting other analysis techniques), theaugmented reality composition system can more intelligently utilizesystem resources. To illustrate, the augmented reality compositionsystem can determine particular techniques most suited to tracking awriting device in poor lighting conditions. In response to detectingpoor lighting conditions, the augmented reality composition system canutilize those particular techniques (and omit other techniques) to trackthe writing device.

More detail regarding the augmented reality composition system will nowbe provided with reference to the figures. For example, FIG. 1illustrates a schematic diagram of an example environment 100 forimplementing an augmented reality composition system 110 in accordancewith one or more embodiments. An overview of the environment 100 isdescribed in relation to FIG. 1. Thereafter, a more detailed descriptionof the components and processes of the augmented reality compositionsystem 110 is provided in relation to the subsequent figures.

As mentioned, FIG. 1 illustrates the environment 100 including server(s)102, a network 104, an augmented reality subsystem 105, and a user 112.The augmented reality subsystem 105 further includes a writing device106 and an augmented reality device 108. The network 104 may be anysuitable network over which computing devices can communicate.Additional detail regarding the network 104 is provided below inrelation to FIG. 9.

As just discussed, the environment 100 includes an augmented realitysubsystem 105 that includes the augmented reality device 108. As usedherein, the term “augmented reality device” refers to a computing devicethat provides a modified view of the real world (i.e., provides anoverlay of digital content over a real world view). In particular, theterm “augmented reality device” includes a computing device thatdigitally augments a contemporaneous view of the real-world withcomputer-generated sensory input, such as sound, images, video,graphics, or data. For example, an augmented reality device includes anaugmented reality headset or augmented reality glasses that include oneor more lenses or display screens that permit a user to view the realword together with augmented reality objects. An augmented realitydevice can also include a camera, microphone, or other data capturingdevices capable of capturing environmental data. For example, anaugmented reality device can utilize a camera to capture environmentaldata to enable the augmented reality device to properly overlayaugmented reality objects in relation to a real-world view.

To illustrate, in relation to the embodiment of FIG. 1, the augmentedreality device 108 includes an augmented reality headset (such asMICROSOFT HOLOLENS, MAGIC LEAP, SONY SMARTEYEGLASS, GOOGLE GLASS, EPSONBT-350, META2 or others). In other embodiments, the user client devicemay refer to a different type of augmented reality device, such as amobile device or smartphone. Regardless of the particular type ofdevice, in relation to FIG. 1, the augmented reality device 108 isassociated with the user 112 and capable of interfacing with the writingdevice 106 and rendering augmented reality objects in response totracking the writing device 106.

Although not illustrated, in one or more embodiments, the augmentedreality device 108 comprises multiple computing devices. For example,the augmented reality device 108 can include (e.g., interface with) aclient computing device such as a tablet, smartphone, personal computer,or other computing device. To illustrate, the augmented reality device108 can provide augmented reality objects for display, while the clientcomputing device performs various calculations or processes to generatethe augmented reality objects and/or communicate with other componentsof the environment 100.

As used herein, the term “augmented reality object” refers to a digitalitem produced by an augmented reality device that modifies perception(e.g., a view) of the real world. An augmented reality object includesdigital images, digital videos, and/or sound provided by an augmentedreality device as an overlay to a real-world environment. An augmentedreality object also includes a digital mark generated by the augmentedreality composition system 110. As used herein, the term “digital mark”refers to an augmented reality object corresponding to interaction witha writing object. In particular, the term “digital mark” includes anaugmented reality object that reflects interaction between a user,writing device, and/or a real-world surface. To illustrate, the term“digital mark” includes a pixel, line, shape, drawing, character,letter, word, or image provided by an augmented reality device. Forexample, the augmented reality composition system 110 can generate adigital mark in an augmented reality environment and provide the digitalmark for display via the augmented reality device 108 such that itappears (to the user 112) that the digital mark is located on areal-world surface based on the writing device 106 pressing and draggingon the real-world surface.

As used herein, the term “augmented reality environment” refers to arepresentation of augmented reality objects relative to real-worldsurroundings. In particular, the term augmented reality environmentincludes a three-dimensional representation of augmented reality objectsin relation to three-dimensional coordinates corresponding to areal-world environment. The augmented reality composition system 110 canprovide an augmented reality environment for display via an augmentedreality device (e.g., by mapping three-dimensional coordinates to adisplay of the augmented reality device). For instance, the augmentedreality composition system can render augmented reality objects on adisplay of the augmented reality device such that the augmented realityobjects appear to be located at the corresponding three-dimensionalcoordinates of the real-world environment.

As used herein, the term “real-world surface” refers to a face of anobject. In particular, the term “real-world surface” may refer to arounded surface, a planar surface, a rough surface, or a smooth surface.In some embodiments, the term “real-world surface” refers to a curvedsurface such as the side of a pillar or column. In other embodiments,the term “real-world surface” refers to a plane of a physical object ina real-world environment. For example, the term “real-world surface”includes a top of a table, a wall, a door, a shelf, or another surfacein the real world. In some embodiments, a real-world surface may referto a surface that the augmented reality composition system 110determines as acceptable for composing (e.g., a surface that issufficiently flat within a drawing threshold).

As used herein, the term “writing device” refers to an implementconfigured to generate digital marks. In particular, the term “writingdevice” includes a cylindrical implement capable of being held by a user(e.g., in the shape of a pen or pencil) for generating digital marks.For example, in relation to FIG. 1, the writing device 106 may refer toa stylus-shaped device that includes various components therein such as,for example, a pressure sensor, a light (e.g., laser) emitter, anoptical sensor, an IMU, or other sensory components.

The writing device 106 includes various components to assist theaugmented reality composition system 110 to track pressure, orientation,location, and/or movement of the writing device 106. For example, thewriting device 106 can include a pressure sensor, a locator element, amotion detector element, and/or an inertial measurement unit. As usedherein, a “locator element” refers to a component integrated as part of(or otherwise associated with a writing device) that allows theaugmented reality composition system 110 to determine a position of thewriting device. As an example, a locator element includes a reflectiveelement (e.g., infrared tape) on the writing device 106 that isconfigured to reflect a particular range of light wavelengths and thatis recognizable by the augmented reality device 108 (e.g., recognizablefor determining location of the writing device 106. A locator elementalso includes an emitter configured to project a visual indicator (e.g.,a laser or other light) onto a real-world surface, where the visualindicator is recognizable by the augmented reality device 108. In thesame or other embodiments, a locator element may refer to a BLUETOOTHlocator configured to transmit waves within a particular range offrequencies whereby the augmented reality device 108 may determine thelocation of the writing device 106. In one or more embodiments,BLUETOOTH is utilized only for communication between the augmentedreality device 108 and the writing device 106.

Similarly, as used herein, the term “motion detector element” refers toa component of a writing device that is configured to track movement. Inparticular, a motion detector element includes a component integrated inthe writing device that allows the augmented reality composition system110 to determine movement (e.g., translation) of the writing device 106(or a tip of the writing device 106) over time. A motion detectorelement may further determine rotation and/or tilt of the writing device106. The motion detector element can include one or more locatorelements. For example, a motion detector element can include an emitterthat projects a visual indicator, a reflective element, or an opticalsensor, as described in greater detail below.

As used herein, an “inertial measurement unit” or “IMU” refers to adevice that measures and reports information regarding changes inorientation and/or location. In particular, an inertial measurement unitincludes a device that measures and reports specific force, angularrate, and/or magnetic field information using a combination of one ormore accelerometers, gyroscopes, and/or magnetometers. To illustrate, anIMU can detect angular changes in pitch, roll, yaw as well as linearposition changes.

Moreover, as used herein, the term “pressure sensor” may refer to devicethat measures a force (or force per unit area) applied to the writingdevice. In particular, a pressure sensor includes a device that measurespressure applied to a tip of the writing device. For example, a pressuresensor may refer to a type of force collector such as a piezoresistivestrain gauge, a capacitive pressure sensor, an electromagnetic pressuresensor, a piezoelectric pressure sensor, an optical pressure sensor, ora potentiometric pressure sensor. A pressure sensor may detect aninitial contact with a real-world surface, and may further detectvariations in the force (and/or force per unit area) applied to thewriting device 106 on the real-world surface. Additional detailregarding the writing device 106 and its components is provided belowwith reference to FIGS. 6A-6B.

As mentioned, the writing device 106 can communicate with the augmentedreality device 108. For example, the writing device 106 may include atransceiver device such as a BLUETOOTH device, a ZIGBEE device, or aWIFI device operable to facilitate communications between the writingdevice 106 and the augmented reality device 108, bypassing network 104.Likewise, the augmented reality device 108 may also include a BLUETOOTHdevice, a ZIGBEE device, a WIFI device, or other transceiver componentoperable to communicate with the writing device 106 in a similarfashion. For example, the writing device 106 communicates with theaugmented reality device 108, and vice-versa, to transmit and receiveinformation relating to pressure sensor data, optical sensor data,object recognition data, emitter data, IMU data, etc., as will bedescribed in further detail below with reference to the subsequentfigures.

As shown in FIG. 1, the environment may also include the server(s) 102.The server(s) 102 may generate, store, receive, and transmit any type ofdata utilized by the augmented reality composition system 110. In oneexample embodiment, the server(s) 102 comprise content servers. Theserver(s) 102 can also comprise a communication server or a web-hostingserver. For instance, in embodiments where the environment 100 includesthe server(s) 102, the augmented reality composition system 110 may beimplemented in whole or in part by the server(s) 102. In addition, theserver(s) 102 may communicate with the augmented reality subsystem 105via the network 104 using an appropriate communications protocol. Inparticular, the server(s) 102 may communicate via the network 104 withthe writing device 106 to send and/or receive information relating tomotion tracking, sensory input, etc. Likewise, the server(s) 102 maycommunicate with the augmented reality device 108 to share all or partof the processing required to implement the augmented realitycomposition system 110 or else to transmit information received from thewriting device 106 relating to sensory input and/or motion trackinginformation. However, in some embodiments, the environment 100 may notinclude the sever(s) 102, and the augmented reality composition system110 may be implemented within the augmented reality subsystem 105—i.e.,across one or both of the writing device 106 and/or the augmentedreality device 108, without the server(s) 102.

As illustrated by FIG. 1, the augmented reality composition system 110may, in one or more embodiments, be included on the augmented realitydevice 108. Moreover, in one or more embodiments, the augmented realitycomposition system 110 may be implemented by the augmented realitydevice 108 in conjunction with one or more other components of theenvironment 100.

By way of example, the augmented reality composition system 110initially analyzes (e.g., via the augmented reality device 108) areal-world environment to detect surfaces and objects within theenvironment. Based on this analysis, the augmented reality compositionsystem 110 generates (e.g., via the augmented reality device 108) athree-dimensional augmented reality environment that includesthree-dimensional reconstructions of surfaces and objects identifiedwithin the real-world environment. The augmented reality compositionsystem 110 additionally analyzes (e.g., via the augmented reality device108) the detected surfaces and objects to identify any surfaces withinthe real-world environment that are acceptable for composing (e.g., thatare sufficiently large and sufficiently flat or planar). Uponidentifying one or more surfaces that are acceptable for composing, theaugmented reality composition system 110 provides indicators to markthose acceptable surfaces within the user's view of the augmentedreality environment. For example, the augmented reality compositionsystem 110 overlays (e.g., via the augmented reality device 108) a greencheck mark on an acceptable surface that is visible to the user via theaugmented reality device.

Continuing the example, the augmented reality composition system 110detects (e.g., via the writing device 106 and/or the augmented realitydevice 108) that the user intends to compose on a real-world surface(e.g., a real-world surface that is acceptable for composing) byreceiving an input from a pressure sensor on the writing device 106 thatindicates that the writing device 106 is touching the real-worldsurface. Upon detecting the pressure input, the augmented realitycomposition system 110 determines an initial location of the writingdevice 106 by way of sensory information determined from the writingdevice 106 and/or by way of object recognition or other functionalitiesof the augmented reality device 108. The augmented reality compositionsystem 110 further determines changes in location, orientation, tilt,and pressure of the writing device 106 as the user moves the writingdevice along the real-world surface to create an augmented realitycomposition. In response to detecting such changes, the augmentedreality composition system 110 renders digital marks in real time toform an augmented reality object for display to the user via theaugmented reality device 108. Upon detecting an indication that the userhas completed composing an augmented reality object (e.g., by a releaseof pressure on the writing device 106), the augmented realitycomposition system 110 enables the user to manipulate the augmentedreality object within the three-dimensional augmented realityenvironment (e.g., via the augmented reality device 108).

As illustrated by the foregoing example, various components of theenvironment 100 may implement the augmented reality composition system110. Indeed, the writing device 106 may communicate with the augmentedreality device 108 (e.g., directly or via network 104) and may implementall or part of the augmented reality composition system 110. Indeed, insome embodiments, the writing device 106 may include a processoroperable to analyze user input and/or process other aspects of theaugmented reality composition system 110. In this way, the processingload to implement the augmented reality composition system 110 may beshared across the augmented reality device 108 as well as the writingdevice 106 and/or the server(s) 102.

Although FIG. 1 illustrates a particular arrangement of the server(s)102, the network 104, the augmented reality subsystem 105, the writingdevice 106, and the augmented reality device 108, various additional oralternative arrangements are possible. For example, while FIG. 1illustrates the writing device 106 and the augmented reality device 108as part of the augmented reality subsystem 105, where the writing device106 and the augmented reality device 108 communicate directly, bypassingnetwork 104, in at least one embodiment, the writing device 106 and/orthe augmented reality device 108 may be outside the augmented realitysubsystem 105 and may communicate via the network 104.

Similarly, although FIG. 1 illustrates the environment 100 including aparticular number of components, the environment 100 may includeadditional or alternative components. For example, the augmented realitycomposition system 110 may be collaborative, where multiple users mayall contribute to the creation of an augmented reality object within asingle shared augmented reality environment (e.g., a plurality of userscould draw on a table with a writing device and generate a sharedaugmented reality object). In these embodiments, the environment 100 mayinclude multiple user client devices, and the augmented realitycomposition system 110 may be implemented by the server(s) 102 or acrossone or more of the multiple user client devices.

As described above, in one or more embodiments, the augmented realitycomposition system 110 tracks interactions between a user, a writingdevice, and/or a real-world surface and generates an augmented realityobject within an augmented reality environment. For instance, FIG. 2illustrates a representation of the augmented reality composition system110 tracking input by way of the writing device 106 and generating anaugmented reality object 204.

More particularly, FIG. 2 illustrates a first view 210 of a real-worldsurface 202 and a second view 220 of the real-world surface 202.Specifically, the first view 210 is a representation of the real-worldsurface 202 viewed without an augmented reality device (i.e. looking ata table without an augmented reality headset), and the second view 220is a representation of the real-world surface 202 as viewed through anaugmented reality device (i.e., looking at the table with the augmentedreality headset 108). In both views 210 and 220, a hand (e.g., of theuser 112) grips the writing device 106 and moves the writing device 106to “draw” on the real-world surface 202 (i.e., draw an image of aglobe).

As shown, in the first view 210, the real-world surface 202 is empty.Indeed, even though a user presses and drags on the real-world surface202 with the writing device 106, the real-world surface 202 does notinclude any writing. In contrast, with the benefit of the augmentedreality device 108, the second view 220 includes an augmented realityobject 204. In particular, the second view 220 includes the augmentedreality object 204, which corresponds to interactions between thewriting device 106 and the real-world surface 202.

In relation to the second view 220, the augmented reality compositionsystem 110 generates the augmented reality object 204 within anaugmented reality environment and presents the augmented reality object204 to the user 112 via the augmented reality device 108. Specifically,the augmented reality composition system 110 tracks pressure, location,orientation, and movement of the writing device 106 as the user 112moves the writing device 106 along the real-world surface 202.

As shown in the second view 220, as the user 112 moves the writingdevice 106 on the real-world surface 202, the augmented realitycomposition system 110 tracks the orientation, pressure, and/or locationof the writing device 106 and generates digital marks within anaugmented reality environment. Accordingly, as the user 112 moves thewriting device 106 to draw the shape of the globe, the augmented realitycomposition system 110 generates the augmented reality object 204.Indeed, as illustrated in FIG. 2, the user 112 is currently drawing aborder. The augmented reality composition system 110 renders theaugmented reality object 204 to appear to the user 112 as though thegenerated digital mark of the border is coming out of the tip of thewriting device 106—i.e., the augmented reality composition system 110renders the digital marks near the tip of the writing device 106 in realtime or near real time as the user moves the writing device 106 on thereal-world surface 202.

As shown in the second view 220, the augmented reality compositionsystem 110 further renders the augmented reality object 204 as though itis drawn on the real-world surface 202. To elaborate, the augmentedreality composition system 110 analyzes a view of real-worldsurroundings of the user 112 (e.g., that the user 112 observes throughthe augmented reality device 108) to identify the real-world surface202. The augmented reality composition system 110 further analyzes thereal-world surface 202 to identify the slope, size, and other attributesof the real-world surface 202. Based on this analysis, the augmentedreality composition system 110 renders the augmented reality object 204on the second view 220 of the real-world surface 202 that the user 112observes by way of the augmented reality device 108 (e.g., to follow theslope, etc. of the real-world surface).

In addition, the augmented reality composition system 110 renders theaugmented reality object 204 to mimic art supplies or other real-worlddrawing implements such as pens, pencils, paints, crayons, etc. Inparticular, the augmented reality composition system 110 providesoptions for the user 112 to select various colors, textures, or otherfeatures of the digital marks that appear to come from the tip of thewriting device 106. Accordingly, the augmented reality compositionsystem 110 can render pen ink digital marks or paint-like digital marks(e.g., in response to user selection of a pen or paint setting).Likewise, the augmented reality composition system 110 can also renderdigital marks in various colors such as red, blue, green, yellow, etc.

Furthermore, though not illustrated in FIG. 2, the augmented realitycomposition system 110 enables a user 112 to interact with the augmentedreality object 204. In particular, the augmented reality compositionsystem 110 may enable the user 112 to touch, move, resize, adjust, warp,or otherwise manipulate the augmented reality object 204 by way of thewriting device 106. Furthermore, the user 112 may manipulate theaugmented reality object 204 in three dimensions—i.e., the augmentedreality composition system 110 enables the user 112 to move theaugmented reality object 204 off of the real-world surface 202 androtate or otherwise move it in three-dimensional space. Thus, theaugmented reality composition system 110 provides an interactiveinterface by which the user 112 is more immersed in the compositionexperience.

Notably, without the augmented reality composition system 100, thereal-world surface 202 appears unchanged. Indeed, as shown in the firstview 210, the real-world surface 202 does not itself contain any marks.Thus, the user 112 is not dependent on the particular real-world surface202 to generate the augmented reality object 204. In fact, the user 112can move to a new location with a new surface and utilize the newsurface to continue drawing the augmented reality object 204.Specifically, the augmented reality composition system can identify anew real-world surface and project the augmented reality object to thenew real-world surface. Thus, the user is not tied to any particularlocation or drawing surface, but can create or modify augmented realityobjects in nearly any locale.

Although not illustrated in FIG. 2, in some embodiments the augmentedreality composition system 110 renders computer-generated digitalobjects such as a color palette or a menu on the real-world surface 202.In particular, the augmented reality composition system 110 generates acolor palette and renders the color palette in a location on thereal-world surface 202 chosen by the user 112 or else by the augmentedreality composition system 110. Indeed, the augmented realitycomposition system 110 may relocate the color palette based on userinput such as, for example, a tap-and-drag motion by way of the writingdevice 106.

The augmented reality composition system 110 further detects a selectionof a color from the color palette by way of the writing device 106. Forexample, the augmented reality composition system 110 detects that thewriting device 106 makes contact with the real-world surface 202 at alocation where the augmented reality composition system 110 renders aparticular color, whereupon the augmented reality composition system 110changes the color of the output of the writing device 106. As anotherexample, the augmented reality composition system 110 detects (e.g., viaa microphone associated with the augmented reality device 108) a voicecommand from the user 112 to select a color from the color palette.

In addition to a color palette, the augmented reality composition system110 can further render other interactive (e.g., user selectable) digitalobjects such as an augmented reality menu. In particular, the augmentedreality composition system 110 can render an augmented reality menu onthe real-world surface 202 that includes various selectable options formanaging settings, user preferences, or other features. For example, theaugmented reality composition system 110 can provide a selectable optionto save or load an augmented reality object (e.g., load an augmentedreality object onto a new reference surface). The augmented realitycomposition system can detect a selection by the user 112 of a menu itemwithin the augmented reality menu, whereupon the augmented realitycomposition system 110 performs the necessary processes to carry out theselected menu request. Alternatively, the augmented reality compositionsystem 110 detects a voice command to selected a menu option by way of amicrophone on the augmented reality device 108.

As mentioned above, the augmented reality composition system 110 isfurther capable of increasing visibility of the augmented reality object204. In particular, in some embodiments the augmented realitycomposition system 110 increases the brightness, hue, contrast, or otherattribute of the augmented reality object 204 to make the augmentedreality object 204 easier for the user 112 to see. In other embodiments,the augmented reality composition system 110 dims an area around theaugmented reality object 204 by decreasing brightness and/or by changinghue and contrast. For example, the augmented reality composition system110 dims the areas of the real-world surface 202 that do not includepart of the augmented reality object 204. As another example, theaugmented reality composition system 110 dims a periphery of theaugmented reality device 108 so that areas in the center of the viewthrough the augmented reality device 108 are bright and clear whileareas away from the center near the edges of the view through theaugmented reality device 108 are less clear or vibrant.

As mentioned, the augmented reality composition system 110 can alsoproject computer-generated digital objects into the view of the user112. For example, the augmented reality composition system 110 canrender a digital image onto the real-world surface 202. To illustrate,the augmented reality composition system 110 can project a digital imageof a car as an augmented reality object onto a real-world surface. Theuser 112 can then utilize the projection of the digital image as part ofa larger composition (e.g., to trace the car or to draw a person sittingin the car).

In some embodiments, the augmented reality composition system 110renders digital objects chosen by the user 112. For example, theaugmented reality composition system 110 enables the user 112 to selectan object (e.g., a digital image) from a menu of the augmented realitycomposition system 110, or alternatively, the augmented realitycomposition system 110 loads an augmented reality object that the user112 previously composed and saved. Similarly, in some embodiments, theaugmented reality composition system 110 renders a digital version of apicture taken and saved by the user 112 by way of the augmented realitydevice 108 or else taken by another device and uploaded to the augmentedreality composition system 110. Thus, by projecting digital objects inthis way, the augmented reality composition system 110 further providesthe user 112 with digital scenery, which may help the user 112 see moreclearly and compose more accurately.

As mentioned above, the augmented reality composition system 110 canutilize a pressure sensor to determine interaction with a real-worldsurface. For example, FIG. 3A illustrates an example writing device 106in contact with the real-world surface 202. In particular, FIG. 3Aillustrates the writing device 106 including a tip 304, a pressuresensor 302, an eraser 303, and an additional pressure sensor 301.Although FIG. 3A illustrates two different pressure sensors 301 and 302on each respective end of the writing device 106, in some embodiments,the writing device 106 need not include both pressure sensors 301 and302, but instead includes one or the other of the pressure sensors 301and 302.

For example, in relation to FIG. 3A the augmented reality compositionsystem 110 detects contact of the writing device 106 with the real-worldsurface 202 by way of the pressure sensor 302. Indeed, the tip 304 ofthe writing device 106 is configured to move in response to a forceapplied to the writing device 106 by the user 112. For example, as theuser 112 presses the writing device 106 down on the real-world surface202, the tip 304 depresses a certain distance into the writing device106 and contacts the pressure sensor 302. In such embodiments, theaugmented reality composition system 110 detects when the tip 304 of thewriting device 106 is pushed in to contact the pressure sensor 302.Although FIG. 3 illustrates a particular type of pressure sensor, theaugmented reality composition system 110 can utilize any type ofpressure sensor described herein. For example, in some embodiments, theaugmented reality composition system 110 can detect a change in pressurewithout movement of the tip 304.

Based on a change in pressure, (e.g., by detecting an initial contact ofthe tip 304 with the pressure sensor 302), the augmented realitycomposition system 110 can determine that the user has begun drawing onthe real-world surface 202. Furthermore, the augmented realitycomposition system can determine an initial location for drawing via thewriting device 106 (e.g., an initial location of the tip 304). Indeed,as mentioned above, to determine the initial location of the writingdevice 106, the augmented reality composition system 110 utilizes one ormore of a number of techniques, including object recognition techniquesor one or more locator elements (e.g., an emitter, reflective material,or other element as described in greater detail below).

In addition to detecting an initial contact of the writing device 106 onthe real-world surface 202, the augmented reality composition system 110further detects an amount of pressure (e.g., a magnitude of thepressure) applied to the writing device 106. Based on changes in thepressure applied to the writing device 106 by the user 112, theaugmented reality composition system 110 renders digital marks withdifferent attributes. To illustrate, when the user 112 is pressing hardon the writing device 106, the augmented reality composition system 110may generate digital marks (e.g., lines) with a heavier weight—i.e., astrength, heaviness, or darkness of lines or marks—or may alternativelygenerate digital marks with a thicker width. Similarly, when the user112 is pressing softly on the writing device 106, the augmented realitycomposition system 110 renders digital marks with a softer weight (e.g.,a lighter appearance) or a thinner width.

In some embodiments, the augmented reality composition system 110detects an initial contact of the writing device 106 and thereafterignores pressure sensor data relating to different amounts of pressureapplied to the writing device 106 while the user 112 composes. In theseembodiments, the augmented reality composition system 110 can apply aconsistent weight that the user 112 selects by way of a selectableoption within the augmented reality environment or by way of a voicecommand detected via a microphone on the augmented reality device 108.

In these or other embodiments, the augmented reality composition system110 changes the weight of rendered digital marks based on detectingwhether the pressure applied to the writing device 106 exceeds aparticular pressure threshold. Indeed, the augmented reality compositionsystem 110 may use multiple pressure thresholds in a tier-based systemwhere, for example, the augmented reality composition system 110 rendersdigital marks having a particular weight while the pressure applied tothe writing device 106 is within a particular pressure range, and wherethe augmented reality composition system 110 renders digital markshaving a different weight while the pressure applied to the writingdevice 106 is within a different range.

In addition, the augmented reality composition system 110 may adjustother compositional attributes based on different pressures applied tothe writing device 106. As used herein, the term “compositionalattribute” refers to a visual characteristic of an augmented realityobject. In particular, a compositional attribute includes a visualcharacteristic of an augmented reality object that the augmented realitycomposition system 110 can modify (e.g., based on changes pressure ortilt of a writing device). For example, a compositional attribute caninclude a weight, thickness, pigmentation, contrast, darkness, oropacity of an augmented reality object.

To illustrate, in response to detecting that the user 112 is applying apressure to the writing device 106 that exceeds a pressure threshold,the augmented reality composition system renders digital marks that aredarker in color (e.g., blacker, a deeper shade of red, or a more intensecoloration of whatever color the augmented reality composition system110 is currently using). As another example, the augmented realitycomposition system 110 may render digital marks with a higher opacity(i.e., lines that are less transparent) in response to detecting thatthe pressure applied to the writing device 106 exceeds a pressurethreshold.

In some embodiments, the augmented reality composition system 110executes a custom command (e.g., as customized by the user 112) inresponse to detecting a pressure applied to the writing device 106 thatexceeds a pressure threshold. For example, the augmented realitycomposition system 110 detects, by way of the pressure sensor 302, ahard tap that exceeds a threshold, and in response, the augmentedreality composition system 110 provides, for display within theaugmented reality environment, a shortcut menu including one or moreuser-selectable options. The augmented reality composition system 110may execute other custom commands in response to detecting a pressurethat exceeds a pressure threshold, as set by the user 112 via, forexample, a custom command menu.

Similar to how the augmented reality composition system 110 detectspressure applied to the pressure sensor 302 of the writing device 106 toadd digital marks, the augmented reality composition system 110 furtherdetects pressure applied to the pressure sensor 301 of the writingdevice 106 to remove digital marks. To illustrate, the augmented realitycomposition system 110 detects variations of pressure applied by theuser 112 to depress the eraser 303 on the back end of the writing device106 into the pressure sensor 301. As the user 112 moves the writingdevice 106 while holding pressure on the eraser 303, the augmentedreality composition system 110 tracks the movement and removes ordeletes those digital marks over which the user 112 moves the eraser 303of the writing device 106. The augmented reality composition system 110can remove digital marks of any size (depending on user selection)including an entire augmented reality object. In other embodiments, theaugmented reality composition system 110 does not include the eraser 303but can remove digital marks via the tip 304 (e.g., upon user selectionof an eraser setting for the tip 304).

In addition to modifying digital marks based on changes in detectedpressure, the augmented reality composition system 110 can also utilizethe pressure sensor 302 to detect a selection (e.g., selection of areal-world surface or an augmented reality object via a double-tapdetected via a pressure sensor). For instance, in a situation where theuser 112 composes more than one digital object, the augmented realitycomposition system 110 can detect a selection of one augmented realityobject or the other via input received by way of the pressure sensor302. Thus, the augmented reality composition system 110 enables the user112 to edit an augmented reality object at a later time, and furtherenables the user 112 to go back and forth between augmented realityobjects within an augmented reality environment. In addition to editing,the augmented reality composition system 110 further enables the user112 to move or otherwise manipulate an augmented reality object inthree-dimensional space (e.g., within the three-dimensional augmentedreality environment).

Furthermore, the augmented reality composition system 110 also utilizesthe pressure sensor 302 to detect a selection of a digital object otherthan a user-created augmented reality object (e.g., augmented realityobject 204). As mentioned above, in some embodiments the augmentedreality composition system 110 renders a color palette or other userselectable menu object on a real-world surface 202. The augmentedreality composition system can also utilize the pressure sensor todetect selection of the color pallet or other selectable menu option.For example, in response to detecting contact by way of the pressuresensor 302 at a location on the real-world surface 202 corresponding toa particular color of a color palette, the augmented reality compositionsystem 110 can change the color of the output of the writing device 106.Likewise, in response to detecting contact via the pressure sensor 302at a location where a given menu item is rendered on the real-worldsurface 202 the augmented reality composition system 110 performs therequisite processes to carry out a menu selection.

As mentioned above, in one or more embodiments, the augmented realitycomposition system 110 can utilize an emitter element to determinelocation and/or movement of a writing device. FIG. 3B illustrates anexample augmented reality device 108 in communication with an examplewriting device 106 with an emitter 306. In particular, FIG. 3Billustrates that the emitter 306 is configured to project a visualindicator 307 onto the real-world surface 202. The augmented realitydevice 108 can detect the location of the writing device 106 byidentifying the projected visual indicator 307 on the real-world surface202.

The emitter 306 can project a variety of different visual indicators.For instance, in some embodiments, the emitter 306 may include a laseremitter that projects the visual indicator 307 in the form of a laserprojection onto the real-world surface 202. In other embodiments, theemitter 306 can project a different light form onto the real-worldsurface 202.

The augmented reality composition system 110 utilizes the visualindicator 307 to determine location of the writing device 106. Forexample, the augmented reality composition system 100 can perform objectrecognition techniques to identify the visual indicator 307 on thereal-world surface 202. In particular, the augmented reality compositionsystem 110 can utilize a plurality of cameras affixed to the augmentedreality device 108 to capture video feeds portraying the real-worldsurface 202 and the writing device 106 from different viewpoints. Theaugmented reality composition system 110 can analyze the video feeds todetermine a location of the visual indicator 307. Specifically, theaugmented reality composition system 110 can analyze the visualindicator 307 portrayed in digital images of the video feeds frommultiple viewpoints and determine a location of the visual indicator 307utilizing various triangulation and/or three-dimensional visual analysistechniques. For example, in one or more embodiments, the augmentedreality composition system 110 implements a Simultaneous Localizationand Mapping (“SLAM”) algorithm to map objects in three-dimensional spacebased on image data. Additional detail regarding three-dimensionalmodeling and image analysis techniques is provided below with referenceto FIG. 4.

Based on the location of the visual indicator 307, the augmented realitycomposition system 110 can also determine a location of the writingdevice 106. For example, the augmented reality composition system 110can determine an offset from the location of the visual indicator 307and the tip 304 of the writing device 106. Thus, the augmented realitycomposition system 110 can apply the offset to the location of thevisual indicator 307 to determine the location of the tip 304 of thewriting device 106. In one or more embodiments, the augmented realitycomposition system 110 utilizes the position of the tip 304 of thewriting device 106 to render digital marks that appear to emerge fromthe tip 304 of the writing device 106.

For example, the augmented reality composition system 110 determines adistance or offset between the visual indicator 307 and the tip 304 ofthe writing device 106 where it is in contact with the real-worldsurface 202. The offset between the visual indicator 307 and the tip 304of the writing device 106 can vary, however, depending on theorientation of the writing device 106. Accordingly, in one or moreembodiments, the augmented reality composition system 110 determines aposition of the tip 304 of the writing device 106 based on both theposition of the visual indicator 307 and a detected orientation of thewriting device 106. For example, the augmented reality compositionsystem 110 can utilize an IMU to determine an orientation of the writingdevice 106, calculate an offset between the visual indicator 307 and atip 304 of the writing device 106 and then determine a location of thetip 304 of the writing device 106. Accordingly, the augmented realitycomposition system 110 determines an offset (e.g., a vector) from thelocation of the visual indicator 307 to the location of the tip 304 ofthe writing device 106. Based on this information, the augmented realitycomposition system 110 renders augmented reality objects within anaugmented reality environment from a location of the tip 304 of thewriting device 106 instead of from the location of the visual indicator307.

Although FIG. 3B illustrates a single visual indicator 307 comprising asingle point, the augmented reality composition system 110 can utilizemultiple visual indicators and/or visual indicators reflecting differentdesigns or shapes. For instance, in one or more embodiments, theaugmented reality composition system 110 utilizes two visual indicators,which increases both the likelihood that the augmented realitycomposition system 110 can detect at least one of the visual indicatorsas well as the accuracy of tracking the writing device 106. Theaugmented reality composition system 110 can utilize the two visualindicators to accurately identify a rotation of the writing device 106and a location of the tip 304 of the writing device 106. Specifically,the augmented reality composition system 110 can determine a location ofa tip 304 of the writing device 106 by analyzing an intersection of twooffsets from two different visual indicators.

Similarly, in one or more embodiments, the augmented reality compositionsystem 110 utilizes a visual indicator of a specific shape thatindicates an orientation of the writing device 106. For instance, theemitter 306 can project a visual indicator 307 in a shape that pointstoward the tip 304 of the writing device 106. In such an embodiment, theaugmented reality composition system 110 can apply an offset in thedirection indicated by the shape to identify the position of the tip 304of the writing device 106.

The augmented reality composition system 100 can utilize the visualindicator 307 to determine an initial location of the writing device 106and/or to track movement of the writing device 106. Thus, the emitter306 can comprise both a locator element and a motion detector element.Indeed, as illustrated in FIG. 3B, the augmented reality compositionsystem 110 can utilize the augmented reality device 108 to obtain videofeeds of the writing device 106 and the visual indicator 307 as thewriting device 106 moves across the real-world surface 202. Theaugmented reality composition system 110 can then utilize the visualindicator 307 to dynamically determine the location of the writingdevice 106.

In this manner, the augmented reality composition system 110 can trackthe movement of the writing device 106 while the writing device 106 isin contact with the real-world surface 202. For example, the augmentedreality composition system 110 determines, by way of the pressure sensor302, that the writing device is in contact with the real-world surface202, and based on that determination, tracks the movement of the writingdevice 106 by tracking changes in location of the visual indicator 307.

In one or more embodiments, the augmented reality composition system 110refrains from tracking the movement of the writing device 106 when thepressure sensor 302 is not engaged—i.e., when the augmented realitycomposition system 110 detects that the writing device 106 is no longerin contact with the real-world surface 202. For instance, in someembodiments, the augmented reality composition system 110 only engagesthe emitter 306 to project the visual indicator 307 upon determining achange in pressure indicative of contact with the real-world surface202. The augmented reality composition system 110 can then utilize theindicator 307 to track the location of the writing device 106.

In other embodiments, on the augmented reality composition system 110tracks the movement of the writing device 106 (e.g., relative to itsinitial location on the real-world surface 202) regardless of whetherthe writing device 106 remains on the real-world surface 202. Toillustrate, in these embodiments, the augmented reality compositionsystem 110 detects the initial location of the writing device 106 asdescribed above, and further tracks the location of the visual indicator307 in three dimensions, even when the user 112 lifts the writing device106 off of the real-world surface 202.

As mentioned above, in addition to utilizing the emitter 306, theaugmented reality composition system 110 can also utilize reflectiveelements to determine location and/or movement of a writing device. Forexample, FIG. 3C illustrates the augmented reality device 108determining the location of the writing device 106 by way of reflectiveelements 308 a and 308 b (referred to herein collectively as “reflectiveelements 308”).

The reflective elements 308 are configured to reflect a particular rangeof light wavelengths. For example, in some embodiments, the reflectiveelements 308 are configured to reflect infrared light which isdetectable by the augmented reality device 108 via an infrared camera orother sensor. In other embodiments, the reflective elements 308 reflectother wavelengths, such as one or more bands of the visible colorspectrum or ultra-violet light wavelengths.

The augmented reality composition system 110 utilizes the reflectiveelements 308 to determine a position of the writing device 106. Forexample, in one or more embodiments, camera devices and/or sensorsaffixed to the augmented reality device 108 capture images of thewriting device 106 and the reflective elements 308. The augmentedreality composition system 110 can then utilize images of the writingdevice 106 and the reflective elements 308 to determine an accuratelocation of the writing device 106.

For example, the augmented reality composition system 110 can utilizevarious triangulation and/or visual analysis techniques (e.g., SLAManalysis, depth mapping, etc.) in conjunction with images portraying thereflective elements 308 from the augmented reality device 108 todetermine the location of the reflective elements 308 on the writingdevice 106. The augmented reality composition system 110 can thenutilize the location of the reflective elements 308 to determine aposition of any portion of the writing device 106 (e.g., a tip 304 ofthe writing device 106). For example, the augmented reality compositionsystem 110 can identify a known differential (distance and/or direction)between the reflective elements 308 and the tip 304 of the writingdevice 106. Upon determining the location of the reflective elements308, the augmented reality composition system can apply the knowndifferential to determine the tip 304 of the writing device 106. In thismanner, the augmented reality composition system 110 can determine aposition (e.g., the initial position) of the writing device 106.

The augmented reality composition system 110 may further utilize thereflective elements 308 to track movement of the writing device 106. Asmentioned, in some embodiments the augmented reality composition system110 tracks the movement of the writing device 106 only while the writingdevice 106 is in contact with the real-world surface 202, while in otherembodiments the augmented reality composition system 110 tracks themovement of the writing device 106 regardless of whether the writingdevice 106 is in contact with the real-world surface 202. The augmentedreality composition system 110 can track the location of the writingdevice 106 utilizing the reflective elements 308 in either circumstance.

By tracking the reflective elements 308, the augmented realitycomposition system 110 determines changes in location of the writingdevice 106, and can thereby render augmented reality objects based onthe movement of the writing device 106. Moreover, the augmented realitycomposition system 110 can generate augmented reality objects thatappear to be drawn by the tip 304 of the writing device 106.

Besides tracking the location of the writing device 106, in someembodiments the augmented reality composition system 110 determines atilt of the writing device 106 by way of the reflective elements 308. Toillustrate, in embodiments where the writing device 106 includes two ormore reflective elements 308, the augmented reality composition system110 calculates a slope or angle between the reflective elements 308 todetermine a tilt of the writing device 106. Based on the tilt of thewriting device 106, the augmented reality composition system 110 mayalter various attributes of the augmented reality objects. For instance,the augmented reality composition system 110 may adjust a thickness orweight of the digital marks based on a tilt of the writing device. Thus,for example, the augmented reality composition system 110 can generate athicker shading digital mark upon detecting a greater tilt of thewriting device 106.

While FIG. 3C illustrates the writing device 106 include two reflectiveelements 308, in some embodiments the writing device 106 may includemore or fewer reflective elements 308. Furthermore, the writing device106 can include reflect elements in different locations with differentshapes. Indeed, in one or more embodiments, the reflective elements 308include markings, variations in shape, or other indications reflectingan orientation of the writing device 106 (e.g., markings indicating whatportion of the writing device 106 is facing toward a camera of theaugmented reality device 108). Thus, the augmented reality compositionsystem 110 can also determine an orientation of the writing device basedon the reflective elements 308.

As mentioned previously, in addition to reflective elements, theaugmented reality composition system 110 may also track movement of awriting device by way of object recognition techniques. For example,FIG. 3D illustrates the augmented reality device 108 performing objectrecognition to identify and track the movement of the writing device106. In particular, the augmented reality device 108 captures (via oneor more camera devices affixed to the augmented reality device) imagesthat portray the writing device 106. The augmented reality compositionsystem 110 can analyze the digital images to identify the writing device106 from other objects portrayed in the digital images.

The augmented reality composition system 110 can utilize a variety ofobject recognition algorithms to identify the writing device 106. Forexample, in one or more embodiments, the augmented reality compositionsystem 110 utilizes pose clustering, interpretation trees, geometrichashing techniques, scale-invariant feature transforms (“SIFT”),speeded-up robust features (“SURF”) algorithms, or bag of words computervision models to identify the writing device 106.

Upon identifying the writing device 106, the augmented realitycomposition system 110 can also determine location and/or movement ofthe writing device 106. For example, as discussed above, the augmentedreality composition system 110 can utilize images portraying the writingdevice 106 to determine a location of the writing device 106.Specifically, the augmented reality composition system 110 can utilizetwo images portraying the writing device 106 from two cameras totriangulate a location of the writing device 106. Indeed, as describedin further detail below with reference to FIG. 4, the augmented realitycomposition system 110 determines a lateral position of the writingdevice 106 relative to the augmented reality device 108 (e.g., withinthe view of the user 112 via the augmented reality device 106) andfurther determines a depth of the writing device 106 relative to theaugmented reality device 108.

As discussed above, in one or more embodiments, the augmented realitycomposition system 110 can also utilize an optical sensor fordetermining location of the writing device 106. For example, FIG. 3Eillustrates the writing device 106 with in optical sensor 310 incommunication with the augmented reality device 108. In particular, FIG.3E illustrates that the writing device 106 includes an optical sensor310 with a light source (e.g., a laser, a light-emitting diode (“LED”)light, or other light). The augmented reality composition system 110tracks movement of the writing device 106 by way of the optical sensor310.

Specifically, as shown in FIG. 3E, the optical sensor 310 projects alight ray 312 from the light source. Moreover, the optical sensor 310detects reflected light (from the light ray 312) off of the real-worldsurface 202. Specifically, the optical sensor 310 detects a degree ofdeflection of the light upon receiving a reflected light off of thereal-world surface 202. The degree of deflection changes as the writingdevice 106 moves on the real-world surface 202. Based on the amount ofdeflection, the augmented reality composition system 110 can determine achange in location of the writing device 106.

Thus, in one or more embodiments, the augmented reality compositionsystem 110 determines an initial location of the writing device 106 andthen utilizes the optical sensor 310 to determine a change in locationfrom the initial location. Specifically, the augmented realitycomposition system 110 utilizes the optical sensor 310 to determine anamount of deflection, calculate a change in location based on the amountof deflection, and then determine a new location of the writing device106 by applying the change in location to the initial location.

In some embodiments, the optical sensor 310 may refer to anoptoelectronic sensor configured to capture successive images of thereal-world surface 202 to track changes in location of the writingdevice 106 via digital image correlation techniques. In someembodiments, the optical sensor 310 captures standard RGB images, whilein other embodiments (e.g., where the texture of a surface is homogenousor difficult to track differences in successive images), the opticalsensor 310 projects invisible patterns onto a surface and capturessuccessive images of the projected patterns. More specifically, theoptical sensor 310 captures an initial image at an initial location ofthe writing device 106 and captures additional subsequent images as thewriting device 106 moves along the real-world surface 202. The writingdevice 106 (and/or the augmented reality device 108) may include aprocessor to analyze and compare the captured images to determine whichdirection and how fast the writing device 106 is moving along thereal-world surface 202 (e.g., by extracting common features from thecaptured digital images and determining the location of the commonfeatures as the writing device 106 moves). Accordingly, the augmentedreality composition system may track relative motion of the writingdevice 106 by translating actual movement of the writing device 106 onthe real-world surface 202 into output (or erasures) of digital markswithin the augmented reality environment.

In these or other embodiments, the writing device 106 includes one ormore processors, a memory, and other components necessary to process andtrack the movement of the writing device 106. For example, the writingdevice 106 can utilize one or more processors to determine an amount ofdeflection and calculate a change in location of the writing device 106.In addition to a processor, the writing device 106 may also include atransceiver element configured to transmit location and/or movementinformation to the augmented reality device 108 by way of, for example,a BLUETOOTH device.

As discussed above, the augmented reality composition system 110 candetermine an orientation of a writing device. For example, FIG. 3Fillustrates the augmented reality composition system 110 determining atilt angle of the writing device 106 on the real-world surface 202. Inparticular, FIG. 3F illustrates a tilt angle θ between the real-worldsurface 202 and the writing device 106.

The augmented reality composition system 110 can determine the tiltangle θ utilizing a variety of different techniques. To elaborate, inrelation to FIG. 3F, the augmented reality composition system 110determines the tilt angle θ between the writing device 106 by way of anIMU (e.g., by determining an amount of tilt via a gyroscope), one ormore reflective elements 308 (e.g., by determining a difference inlocation between two reflective elements and calculating an angle basedon the difference in location), and object recognition techniques. Bydetermining the tilt angle θ, the augmented reality composition system110 more accurately tracks the writing device 106. For instance, theaugmented reality composition system 110 can extrapolate a location ofthe tip 304 of the writing device 106 in situations where the augmentedreality device 108 cannot directly recognize or “see” the tip 304 of thewriting device 106 (e.g., when the user's 112 hand is blocking theview).

To illustrate, based on a predefined length, width, and/or otherdimensions of the writing device 106, and further based on the tiltangle θ, the augmented reality composition system 110 may calculate thelocation of tip 304 from any given identified portion of the writingdevice 106. Accordingly, the augmented reality composition system 110may generate augmented reality objects by generating digital marks froma location of the tip 304 of the writing device 106 within the augmentedreality environment, even when the user 112 cannot see the tip 304 ofthe writing device 106 through the augmented reality device 108 due toan obstruction.

In addition, the augmented reality composition system 110 can adjustattributes of augmented reality objects (i.e., adjust digital marks)based on various values for the tilt angle θ. As discussed above, agreater tilt angle θ may result in thicker lines, darker lines, and/ormore opaque lines, while a lesser tilt angle θ may result in thinnerlines, lighter lines, or less opaque lines.

Although FIGS. 3A-3F discuss various techniques (generally in isolation)for tracking the movement, orientation, tilt, and pressure of thewriting device 106, in one or more embodiments the augmented realitycomposition system 110 may incorporate two or more of the aforementionedtechniques to track the writing device 106. Accordingly, the writingdevice 106 may include one or more of the above-mentioned features orcomponents, as will be described in further detail below (e.g., withreference to FIGS. 5, 6A-6B).

Though not illustrated in FIGS. 3A-3F, the augmented reality compositionsystem 110 may additionally or alternatively utilize a trained machinelearning model to track movement of the writing device 106. Inparticular, in circumstances where the tip of the writing device 106 isdifficult to locate (or in other circumstances), the augmented realitycomposition system 110 can gather visual cues such as hand position ofthe user 112, grip shape on the writing device 106, tilt of the writingdevice 106, previous movements of the writing device 106, etc. Based onthese cues, the augmented reality composition system 110 utilizes amachine learning model such as a convolutional neural network to predictmovement of the writing device 106.

For example, the augmented reality composition system 110 can train amachine learning model by providing digital images and/or video of auser drafting a composition utilizing a writing device. The augmentedreality composition system 110 can also track ground-truth informationregarding location, orientation, tilt, and/or movement of the writingdevice. The augmented reality composition system 110 can provide thedigital images and/or video to the machine learning algorithm andgenerate a prediction of location, orientation, tilt and/or movement.The augmented reality composition system 110 can then compare theprediction to the ground truth information (e.g., the actual location,orientation, tilt, and/or movement). Specifically, the augmented realitycomposition system 110 can determine a loss function between theprediction and ground truth information. The augmented realitycomposition system 110 can then utilize the loss function to train theneural network to more accurately predict location, orientation, tilt,and/or movement of a writing device based on images of a user (e.g.,images of a user's grip, arm position, etc.).

As mentioned above, in one or more embodiments, the augmented realitycomposition system 110 can identify real-world surfaces from areal-world environment. For example, FIG. 4 illustrates an example view400 of a real-world environment through the augmented reality device 108in accordance with one or more embodiments. As shown, the real-worldenvironment includes a table, chairs, television, and walls, withvarious real-world surface 402-408.

To analyze the room illustrated in FIG. 4, the augmented realitycomposition system 110 implements a three-dimensional modelingalgorithm. The augmented reality composition system 110 can utilize anumber of different three-dimensional modeling algorithms, includingvarious depth triangulation techniques and/or visual analysistechniques. In relation to the embodiment of FIG. 4, the augmentedreality composition system 110 utilizes Simultaneous Localization andMapping (“SLAM”) technologies. With SLAM techniques, the augmentedreality composition system 110 can establish correlations between colorvalues (e.g., RGB image values) and depths/positions those RGB valuesindicate in three-dimensional space. Accordingly, the augmented realitycomposition system 110 reconstructs a three-dimensional environment(e.g., an augmented reality environment reflecting the view 400) torepresent the three-dimensional positions ascertained by way of RGBvalues. Based on the three-dimensional reconstruction, the augmentedreality composition system 110 detects primitive shapes such as, forexample, planar surfaces, cuboids, or other relatively large, flatsurfaces acceptable for composing.

As just mentioned, in some embodiments, the augmented realitycomposition system 110 utilizes one or more depth mapping techniquesthat utilize views from multiple cameras (e.g., cameras associated withthe augmented reality device) to ascertain depths of surfaces andobjects in three-dimensional space. Additionally or alternatively, theaugmented reality composition system 110 utilizes depth image basedrendering (“DIBR”) techniques to analyze two-dimensional images or video(e.g., as captured by a single camera) to generate three-dimensionalapproximations of depth for various objects and surfaces within thetwo-dimensional images. DIBR techniques can include, but are notnecessarily limited to, implementing a machine learning model trained toanalyze images and determine depths of each portion of an image relativeto a camera perspective.

In one or more embodiments, the augmented reality composition system 110utilizes one or more depth sensors. In such cases, a depth sensorincludes, but is not necessarily limited to, an infrared projector andan infrared camera. The infrared projector projects infrared dots (orother output) which the infrared camera captures. Based on the patternand displacement of the dots that the infrared camera captures, theaugmented reality composition system 110 calculates relative depth ofobjects within a given camera view of a real-world environment. In othercases, a depth sensor additionally includes an RGB camera that capturescolor values of objects within the real-world environment to increaseaccuracy of depth calculations. For instance, changes in hue, contrast,or other color values can indicate varying depths withinthree-dimensional space.

The augmented reality composition system 110 analyzes the room by way ofthe augmented reality device 108 to identify those surfaces within theroom that are acceptable for composing. To determine which of thereal-world surfaces 402-408 are acceptable for composing, the augmentedreality composition system 110 may perform any of a number of spatialrecognition techniques. For example, the augmented reality compositionsystem 110 may analyze each surface in the room to determine whichsurfaces are sufficiently smooth (and/or within a maximum level ofbrightness). To illustrate, the augmented reality composition system 110analyzes each surface to identify those surfaces that are smooth enough(e.g., above a smoothness threshold) and/or non-reflecting enough (e.g.,within a threshold level of brightness or reflectivity) where the user112 could feasibly compose.

Moreover, the augmented reality composition system 110 may analyze eachsurface in the room to determine which surfaces are sufficiently largeand/or sufficiently flat for the user 112 to compose. The augmentedreality composition system 110 may identify those surfaces within theview of the room that are above a certain size (e.g., surface area)threshold and/or that are above a certain planar threshold (e.g., athreshold measuring how smooth or flat a surface appears). The augmentedreality composition system 110 may analyze each surface forimperfections, bumps, slopes, gaps, uneven portions, etc., to determinewhether a given surface meets or exceeds a planar threshold.

Based on analyzing the room to identify acceptable surfaces, theaugmented reality composition system 110 renders, within the augmentedreality environment 400, an indicator for each of the real-worldsurfaces 402-408 (e.g., to designate the real-world surfaces asacceptable or not acceptable). In some embodiments, the augmentedreality composition system 110 may only designate acceptable surfaces(and may not designate those surfaces that are unacceptable). Forpurposes of illustration, however, the exemplary embodiment of FIG. 4indicates those surfaces that are acceptable for composing and thosethat are not.

As shown in FIG. 4, the augmented reality composition system 110analyzes the real-world environment and determines that real-worldsurface 402 is acceptable for composing. In particular, the augmentedreality composition system 110 implements one or more of thethree-dimensional modeling techniques described above. For instance, insome cases, the augmented reality composition system 110 utilizes a SLAMmodel to analyze the room illustrated in FIG. 4 to identify thosesurfaces within the room that are potentially viable for composing.Together with the SLAM model (or other three-dimensional modelingtechnique), the augmented reality composition system 110 applies athreshold determination to identify those potentially viable surfaceswithin the room that also satisfy a planar threshold. The augmentedreality composition system 110 then generates indicators to label thosesurfaces as acceptable for composing.

To illustrate, the augmented reality composition system 110 determinesthat the real-world surface 402 is sufficiently flat and sufficientlylarge for composing. Based on this determination, the augmented realitycomposition system 110 designates the real-world surface 402 asacceptable for composing by way of a visual indicator. The visualindicator can take a variety of forms. As shown, the visual indicatormay include a check mark, a color overlay (e.g., color shading of thesurface), a pattern overlay, and/or a texture overlay for the real-worldsurface 402. The visual indicator can also have a variety of visualeffects, such as flashing, glowing, and/or changing colors. Accordingly,the augmented reality composition system 110 generates the visualindicator within the augmented reality environment 400 and presents thevisual indicator for display to the user 112 within the augmentedreality environment 400 as an overlay of the real-world surface 402.

As illustrated in FIG. 4, the augmented reality composition system 110further renders an indicator for real-world surface 404. In particular,the augmented reality composition system 110 analyzes the room depictedin FIG. 4 to determine that the real-world surface 404 is unacceptablefor composing. For example, the augmented reality composition system 110analyzes the real-world surface 404 and determines that the real-worldsurface 404 does not satisfy a planar threshold, as described above. Toillustrate, the augmented reality composition system 110 detects thegrooves within real-world surface 404 as illustrated in FIG. 4, andtherefore determines that real-world surface 404 does not satisfy theplanar threshold and is unacceptable for composing.

Based on determining that real-world surface 404 is unacceptable forcomposing, the augmented reality composition system 110 renders a visualindicator to designate the real-world surface 404 as unacceptable. Theaugmented reality composition system 110 can utilize a variety of visualindicators to indicate an unacceptable surface. For example, a visualindicator may include an “x.” Similarly, as mentioned above, the visualindicator can include a particular color, texture, pattern, or effect.In this manner, the augmented reality composition system 110 generatesthe visual indicator and renders the visual indicator as an overlay ofthe real-world surface 404 within the augmented reality environment 400.

The augmented reality composition system 110 may identify multipleacceptable surfaces within a single real-world environment. Forinstance, similar to the discussion above with reference to real-worldsurface 402, the augmented reality composition system 110 determinesthat the real-world surface 406 is also acceptable for composing.Indeed, as shown in FIG. 4, the augmented reality composition system 110generates visual indicators in the form of check marks and a particularcolor overlay for each of real-world surface 402 and the real-worldsurface 406.

The augmented reality composition system 110 can also identify multiplesurfaces unacceptable for composing. For instance, as shown, theaugmented reality composition system 110 designates real-world surface408 as unacceptable for composing. In particular, the augmented realitycomposition system 110 analyzes the real-world surface 408 anddetermines that the surface is too uneven (e.g., not smooth enoughand/or not large enough) to be a feasible surface on which user 112could compose. Indeed, as shown in FIG. 4, the real-world surface 408contains a number of canned light fixtures, which render the real-worldsurface 408 unacceptable for composing.

Based on determining that a real-world surface is acceptable forcomposing, the augmented reality composition system 110 provides thefunctions and features described herein for enabling a user 112 tocompose within an augmented reality environment by way of a writingdevice 106 on a real-world surface. In contrast, based on determiningthat a real-world surface is unacceptable for composing, the augmentedreality composition system 110 may disable one or more of the functionsand/or features described herein. In other embodiments, however, theaugmented reality composition system 110 may still enable a user 112 tocompose on a surface that is identified as unacceptable. Thus, in theseembodiments, designating a real-world surface (e.g., real-world surface404 or 408) may serve as a warning rather than a prohibition.

In some embodiments, the augmented reality composition system 110enables user 112 to relocate an augmented reality object from onesurface to another. To illustrate, the augmented reality compositionsystem 110 analyzes the room illustrated in FIG. 4 to determine thatreal-world surfaces 402 and 406 are acceptable for composing. Inaddition, the augmented reality composition system 110 provides anoption (e.g., a user selectable digital element within the augmentedreality environment 400 or via detecting a voice command) for the user112 to save an augmented reality object that is currently in progress.The augmented reality composition system 110 further provides an optionfor the user 112 to load a saved augmented reality object to render theaugmented reality object on a different real-world surface. For example,the augmented reality composition system 110 renders the augmentedreality object on a different real-world surface in the same room ofFIG. 4 or else on a new surface in a different place altogether.

In some cases, the user 112 need not save an augmented reality object tomove it to another surface. Rather, the augmented reality compositionsystem 110 detects a user selection of the augmented reality object anddetects a motion to carry, throw, or otherwise move the augmentedreality object from, for example, real-world surface 402 to real-worldsurface 406, whereupon the augmented reality composition system 110renders the augmented reality object as an overlay of the real-worldsurface 406.

Similarly, in some embodiments the augmented reality composition system110 enables a user 112 to pass an augmented reality object to anotheruser. To illustrate, the augmented reality composition system 110detects a selection of an augmented reality object as described above,and the augmented reality composition system 110 further detectsmovement such as a touch-and-drag motion to move the augmented realityobject. In some embodiments, the augmented reality composition system110 detects a selection (e.g., a press-and-hold by way of the writingdevice 106 or a voice command by way of the augmented reality device108) of an augmented reality object and further detects movement of thewriting device 106 to drag the augmented reality object to transfer toanother user.

To successfully transfer an augmented reality object, in someembodiments, the augmented reality composition system 110 requires thatthe users be within a threshold distance of each other or some otherlimiting factor (e.g., requires that user 112 uses the writing device106 to tap the writing device of the other user to indicate a transferof the augmented reality object). In any case, the augmented realitycomposition system 110 transfers (or sends a copy of) the augmentedreality object to the other user by way of an augmented reality deviceof the other user.

Although FIG. 4 illustrates a single view 400, the augmented realitycomposition system 110 may generate different views includingdesignations of surfaces that are acceptable for composing in responseto movement by the user 112. For example, as the user 112 turns to lookat different parts of the room or walks to enter a different room, theaugmented reality composition system 110 may continue to generate andexpand the view 400 to include new surfaces, and may store in memoryinformation pertaining to previously-analyzed surfaces that may or maynot be within the current view of the user 112 as seen via the augmentedreality device 108. To elaborate, the augmented reality compositionsystem 110 may store a three-dimensional location of each identifiedsurface and may further store an indication of whether each identifiedsurface is acceptable or unacceptable for composing within an augmentedreality environment. Thus, as the user 112 returns to previous rooms orturns back to look at surfaces which the augmented reality compositionsystem 110 has already analyzed, the augmented reality compositionsystem 110 may render and overlay the same visual indicators bydisplaying augmented reality objects from the augmented realityenvironment 400.

Indeed, as mentioned previously, an augmented reality environment may bea three-dimensional map of locations, surfaces, objects, etc., that theaugmented reality composition system 110 analyzes, identifies, andstores in memory. Thus, based on detecting movement, rotation, etc., ofthe augmented reality device 108 (e.g., by way of a global positionsystem (“GPS”) device, IMU, etc.) the augmented reality compositionsystem 110 determines distances, angles, and other relationships betweeneach identified surface, object, etc., within a real-world environmentand generates an augmented reality environment. By generating athree-dimensional augmented reality environment, the augmented realitycomposition system 110 can render overlays on each identified surface,object, etc., based on the three-dimensional coordinates of the same.

As discussed above, the augmented reality composition system 110 canutilize different techniques in different circumstances to track awriting device and generate augmented reality objects. For instance, theaugmented reality composition system 110 can utilize a first techniqueto determine an initial location of the writing device 106 on areal-world surface (e.g., real-world surface 202), and can use the sameor other techniques to track movement of the writing device 106. Thisapproach allows the augmented reality composition system 110 to operateefficiently (e.g., safe valuable processing resources) and accurately(e.g., utilizing the most accurate techniques for tracking a writingdevice in a particular circumstance).

For example, FIG. 5 illustrates an example table 500 illustratingvarious techniques that the augmented reality composition system 110 canutilize in different circumstances or situations. For example, the table500 includes a vertical column of different situations in which theaugmented reality composition system 110 can operate. FIG. 5 furtherillustrates a horizontal row of techniques and/or features by which theaugmented reality composition system 110 can perform the variousfunctions described herein.

To detect different circumstances, the augmented reality compositionsystem 110 analyzes writing surfaces (e.g., real-world surface 202),lighting conditions, and other factors of the real-world environment, inaddition to movement, tilt, and other factors associated with thewriting device 106. Depending on the circumstances identified by theaugmented reality composition system 110, the augmented realitycomposition system 110 determines appropriate techniques to implement tomonitor the writing device 106 and render augmented reality objectswithin an augmented reality environment (e.g., augmented realityenvironment 404).

The augmented reality composition system 110 can utilize a variety ofmethods to determine what techniques to utilize under whatcircumstances. For example, in one or more embodiments, the augmentedreality composition system 110 determines a technique based on userselection and/or user preferences. To illustrate, the augmented realitycomposition system 110 may provide user selectable options for a user tochoose what techniques to apply in particular circumstances. Theaugmented reality composition system 110 can store and access suchsettings defined by the user 112 that dictate how the augmented realitycomposition system 110 responds to particular detected circumstances.

In one or more embodiments, the augmented reality composition system 110pre-defines what techniques to utilize in what circumstances. Forinstance, in one or more embodiments, the augmented reality compositionsystem 110 stores the table 500 and, in response to detecting particularcircumstances, applies the techniques indicated in the table 500 (whilerestricting the techniques not indicated).

Alternatively, the augmented reality composition system 110 may utilizea machine learning model or a Kalman filter to determine whichtechniques are more effective (e.g., more accurate or morecomputationally efficient) under what circumstances. To illustrate, theaugmented reality composition system 110 may utilize a machine learningmodel that is trained (e.g., based on past performance as indicated byresponse to user survey or other user input) to identify those trackingtechniques that are effective in particular circumstances (e.g., whattechnique(s) are most effective when the writing device 106 is movingfast versus when the writing device 106 is moving slowly, whattechnique(s) are most effective under good lighting conditions versusbad lighting conditions). Additionally, the augmented realitycomposition system 110 may switch between the various techniquesdescribed in table 500 of FIG. 5, and may further combine one or more ofthe techniques in a given circumstance in accordance with the machinelearning model.

As shown in FIG. 5, the augmented reality composition system 110 mayutilize or exclude techniques or features in response to detectingdifferent circumstances. For instance, as shown in FIG. 5, the augmentedreality composition system 110 may detect fast movement of a writingdevice (e.g., a user quickly scribbling on a writing surface). Sometechniques may be more efficient and/or accurate in tracking fastmovement of a writing device. Indeed, in relation to the embodiment ofFIG. 5, in response to detecting fast movement of a writing device, theaugmented reality composition system 110 tracks the writing device byway of an optical sensor, an IMU, and/or a machine learning technique.

To elaborate, the augmented reality composition system 110 may detect aspeed at which the user 112 moves the writing device 106, and mayfurther determine whether the speed exceeds a speed threshold (e.g.,corresponds to a fast speed or a slow speed). In some embodiments,however, the augmented reality composition system 110 may implementmultiple speed thresholds to delineate between various speeds of thewriting device 106 instead of only determining between two differentspeeds.

When the augmented reality composition system 110 determines that thewriting device is moving fast, the augmented reality composition system110 can implement those techniques which are better suited for detectingfast movement. For example, with regard to FIG. 5, the augmented realitycomposition system 110 tracks fast movement of the writing device 106using one or more of: an optical sensor (as described above withreference to FIG. 3E), an IMU, and/or a machine learning model.

For instance, as discussed above, to track movement of the writingdevice 106 via a machine learning model, the augmented realitycomposition system 110 trains a neural network (e.g., a convolutionalneural network) or other deep learning model based on previous movementsof the user 112, the writing device 106, and/or other a training set ofother users or writing devices. The augmented reality composition system110 further trains the machine learning model based on visual cues suchas arm position of the user 112, hand position, grip shape on thewriting device 106, tilt of the writing device 106, among others. Inaddition, the augmented reality composition system 110 may implement anerror reducing technique in training the model to improve the accuracyof the model in predicting movement. Using a trained model, theaugmented reality composition system 110 can more accurately determinemovements of the writing device 106, especially in situations wheredirect line of sight between the augmented reality device 108 and thewriting device 106 is obstructed, by predicting where the writing device106 is located based on movement by the user 112.

In addition to fast movement, as shown in FIG. 5, the augmented realitycomposition system 110 may also implement one or more of a number oftechniques in response to slow movement of the writing device 106. Forexample, as shown in FIG. 5, in response to detecting slow movement of awriting device, the augmented reality composition system 110 may use areflective element such as infrared tape (as discussed above withreference to FIG. 3C), an emitter (e.g., a laser emitter) operable toproject a visual indicator onto a real-world surface (as discussed abovewith reference to FIG. 3B), an optical sensor (as discussed above withreference to FIG. 3E), and/or object recognition techniques.

As further illustrated by FIG. 5, the augmented reality compositionsystem 110 may also select different techniques in response to detectingthat a writing device is in an initial position for generating a drawingmark. Indeed, in response to detecting a change in pressure thatindicates that a writing device is in contact with a real-world surface,the augmented reality composition system 110 can select one or moretechniques to detect the initial position.

In some embodiments, the augmented reality composition system 110 mayimplement techniques similar to those that the augmented realitycomposition system 110 uses to track slow movement of the writing device106 in order to determine an initial location of the writing device 106.For instance, as shown in FIG. 5, upon detecting initial contact betweena writing device and a real-world surface, the augmented realitycomposition system 110 can utilize a reflective element, an emitter,and/or object recognition techniques (described in detail above) todetermine the initial position.

Indeed, as described above, the augmented reality composition system 110may determine an initial location by detecting a three-dimensionalcoordinate location of one or more reflective elements located on thewriting device via an augmented reality device 108. The augmentedreality composition system 110 determines a three-dimensional coordinateof the reflective element(s) by implementing one or morethree-dimensional modeling techniques (e.g., SLAM analysis, depthmapping, or other three-dimensional image analysis methods) describedabove. In accordance with the three-dimensional analysis, the augmentedreality composition system 110 translates the three-dimensionalreal-world coordinate of the reflective element(s) into athree-dimensional coordinate of the writing device 106 within theaugmented reality environment. In cases where the reflective element(s)are not located immediately on the tip of the writing device 106, theaugmented reality composition system 110 further calculates a distancebetween the reflective element(s) and the tip of the writing device 106in order to determine the initial location of the tip of the writingdevice to render augmented reality objects as described above.

In addition to, or alternatively to, using a technique to detectreflective elements, the augmented reality composition system 110 maydetect a location of a visual indicator projected by an emitterassociated with the writing device 106. As described above, the writingdevice 106 may include an emitter that projects a visual indicator ontoa real-world surface. To detect the initial position of the writingdevice 106 based on the position of a visual indicator projected onto areal-world surface, the augmented reality composition system 110receives information that indicates a position (e.g., a distance and anangle) of the emitter relative to the tip of writing device 106. Basedon this information, the augmented reality composition system 110calculates a distance from the visual indicator projected onto thereal-world surface to the tip of the writing device 106. Accordingly,the augmented reality composition system 110 determines an initialposition of the writing device 106 using the emitter of the writingdevice 106.

As another technique to determine an initial position of the writingdevice 106, the augmented reality composition system 110 may implementone or more object recognition techniques. In particular, the augmentedreality composition system 110 implements a spatial recognitiontechnique such as depth mapping to determine a position of the writingdevice 106 relative to a real-world surface. By using depth mapping orother three-dimensional modeling method, the augmented realitycomposition system 110 determines a three-dimensional coordinatelocation of the writing device 106 on the real-world surface.

As further illustrated in table 500 of FIG. 5, the augmented realitycomposition system 110 may also select one or more techniques inresponse to detecting tilt of a writing device. For example, theaugmented reality composition system 110 can implement techniques thatare most effective at detecting tilt in response to detecting that awriting device exceeds a particular tilt threshold. For instance, inresponse to detecting that a writing devices satisfies a tilt threshold,the augmented reality composition system 110 can track further tilt ofthe writing device by way of reflective elements, object recognitiontechnologies, and/or an IMU.

For example, the augmented reality composition system 110 may determinea tilt of the writing device 106 by way of reflective elementsassociated with the writing device 106. As described above, theaugmented reality composition system 101 may determine an angle betweena first reflective element located near the tip of the writing device106 and a second reflective element near the opposite end of the writingdevice 106.

Similarly, the augmented reality composition system 110 may determine atilt of the writing device 106 via one or more object recognitiontechniques described herein. To illustrate, the augmented realitycomposition system 110 determines three-dimensional coordinate locationof the tip of the writing device 106 and further detects athree-dimensional coordinate location of an eraser or back end of thewriting device 106. In addition, the augmented reality compositionsystem 110 determines a slope between these two locations to calculatethe tilt of the writing device 106.

As an alternative, or in addition, the augmented reality compositionsystem 110 may determine a tilt of the writing device 106 by way of anIMU associated with the writing device 106. To illustrate, the augmentedreality composition system 110 may detect an angle that the user 112 isholding the writing device 106 via an accelerometer and/or gyroscope,and may further detect changes in the angle in a similar fashion.Indeed, the augmented reality composition system 110 receivesinformation from the IMU associated with the writing device 106, whichinformation can include an instantaneous tilt of the writing device 106at any given point in time, and which updates at regular intervals(e.g., every millisecond, every microsecond), or else updates wheneverthe augmented reality composition system 110 queries the writing device106 for tilt information.

As also illustrated in FIG. 5, in addition to tilt, the augmentedreality composition system 110 can also apply different techniques inresponse to different lighting within the real-world environment. Forexample, in some embodiments the augmented reality composition system110 can detect lighting conditions (e.g., via a camera or ambient lightsensor of an augmented reality device 108) and determine one or morelighting conditions (e.g., too dark, too bright, glare, etc.). Upondetecting the lighting conditions the augmented reality compositionsystem 110 can select a technique appropriate to the lightingconditions.

For instance, in one or more embodiments, the augmented realitycomposition system 110 tracks a writing device in areas with badlighting by using techniques that are less susceptible to lightingeffects. For example, in situations where the ambient lighting preventssensors and/or cameras associated with the augmented reality device 108from effectively locating and/or tracking the writing device 106 (e.g.,due to glares, overly bright lights, dim or dark conditions, etc.), theaugmented reality composition system 110 may utilize one or moretechniques to effectively track movement of the writing device 106. Asshown in FIG. 5, in one or more embodiments, the augmented realitycomposition system 110 utilizes a reflective element, an optical sensor,and/or an IMU.

Indeed, to illustrate, in one or more embodiments, the augmented realitycomposition system 110 utilizes reflective elements located on thewriting device 106 to track the writing device 106. For instance, asdescribed above, the reflective elements are configured to reflect aspecific range of light wavelengths such as, for example, infraredlight. Thus, the augmented reality composition system 110 tracks thereflective elements by detecting infrared (or other wavelengths) lightreflected off of the reflective elements by way of an infrared camera orother light sensor configured to capture the particular light reflectedlight wavelengths. Therefore, even in circumstances where lighting isbad (e.g., it is difficult to see due to darkness, glares, etc.),visible light wavelengths that affect the visibility of the writingdevice 106 have little to no effect on an infrared camera detectinginfrared light. Thus, the augmented reality composition system 110 caneffectively track the writing device 106 in bad lighting via reflectiveelements, as illustrated in FIG. 5.

In some embodiments, augmented reality composition system 110 mayadditionally or alternatively implement a technique that uses theabove-described optical sensor to track the writing device 106 in badlighting conditions. In particular, since the optical sensor is locatedon the writing device 106, a processor on the writing device 106 canprocess successive images or other changes in reflected light todetermine relative motion of the writing device 106. The augmentedreality composition system 110 receives the movement information fromthe writing device 106 via BLUETOOTH or some other communication method.Thus, by utilizing an optical sensor, the augmented reality compositionsystem 110 effectively tracks the writing device 106 even in badlighting.

In a similar fashion, in some embodiments, the augmented realitycomposition system 110 tracks the writing device 106 via an IMU of thewriting device 106. As described above, the IMU collects informationsuch as acceleration, position change, relative direction, angle,orientation, and elevation of the writing device 106. The augmentedreality composition system 110 receives this information from thewriting device 106 via BLUETOOTH and thereby calculates movement, tilt,and orientation of the writing device 106. Thus, by using the IMU, theaugmented reality composition system 110 is unaffected by bad lightingin tracking the writing device 106.

As shown in FIG. 5, in situations where the lighting around the writingdevice 106 is good, the augmented reality composition system 110 maytrack movement of the writing device 106 by way of techniques that aremore camera-reliant. For example, the augmented reality compositionsystem 110 may track the writing device by way of one or more reflectiveelements associated with the writing device 106 (which may be operablein good or bad lighting, depending on the particular circumstances) oran emitter (which also may work in good or bad lighting, depending onthe circumstance, such as when the lighting is dark).

As further illustrated in FIG. 5, the augmented reality compositionsystem 110 can also select a technique upon detecting characteristics ofa real-world surface (e.g., a reflective surface). For instance, in oneor more embodiments, the augmented reality composition system 110detects characteristics (e.g., reflectivity) of a surface utilizing acamera affixed to an augmented reality device. To illustrate, theaugmented reality composition system 110 may determine that a surface isreflective by detecting a brightness of light reflected off of thereal-world surface and determining that the brightness meets or exceedsa threshold.

In response to detecting surface characteristics, the augmented realitycomposition system can select techniques for tracking the writing device106 (e.g., by way of techniques that are less sensitive to changes inlight). For instance, as shown in FIG. 5, the augmented realitycomposition system 110 may utilize object recognition techniques, anIMU, or machine learning techniques.

As shown in FIG. 5, the augmented reality composition system 110 canalso detect that a surface is not reflective and select a technique fortracking the writing device 106. For instance, the augmented realitycomposition system 110 determines that the surface is non-reflective bydetecting an intensity or brightness of light reflected off of thereal-world surface and determining that the brightness is below athreshold. In response to determining that a real-world writing surfaceon which the user is composing is non-reflective, the augmented realitycomposition system 110 implements a technique to effectively track thewriting device 106. As shown in FIG. 5, the augmented realitycomposition system 110 tracks the writing device 106 by way of one ormore reflective elements, an emitter, an optical sensor and/or a trainedmachine learning model.

As further illustrated in FIG. 5, the augmented reality compositionsystem 110 may select one or more tracking techniques in response todetecting that all or part of the writing device 106 is blocked fromview (e.g., view of the augmented reality device 108). In particular,the augmented reality composition system 110 may detect when the tip ofthe writing device 106 is obstructed from the view of the augmentedreality device 108 (e.g., when the user's hand or arm is in the way),and may therefore implement one or more of the techniques indicatedwithin table 500 to track the writing device 106. For example, theaugmented reality composition system 110 may track the writing device106 via techniques that are less reliant on a clear line of sight suchas an optical sensor, an IMU, or a machine learning technique.Similarly, where a tip of the writing device is out of view (but otherportions of the writing device are within view), the augmented realitycomposition system 110 may also utilize a reflective element or objectrecognition technique to track the writing device.

Although FIG. 5 illustrates example techniques that are applicable incertain situations, it will be understood that in some embodiments, theaugmented reality composition system 110 may utilize additional oralternative techniques (or additional or alternative combinations) forthe situation listed in FIG. 5. For example, the augmented realitycomposition system 110 may implement a trained machine learning model insituations with good lighting. Additionally, the augmented realitycomposition system 110 may, in some embodiments, determine a tilt of thewriting device 106 by way of object recognition techniques. Indeed, theaugmented reality composition system 110 may implement each of thetechniques and/or features described herein alone or in combination withother techniques/features to determine the initial position of, tilt of,and track movement of, the writing device 106.

Furthermore, the augmented reality composition system 110 may combinetwo or more of the techniques described in relation to FIG. 5. Moreover,the augmented reality composition system 110 can detect multiplecircumstances at a given time and utilize techniques associated witheach of the circumstances. Indeed, in some cases, two or more of thecircumstances described in table 500 will be satisfied at the same timesuch as, for example, fast movement of the writing device 106 on areflective surface in bad lighting. Accordingly, the augmented realitycomposition system 110 determines which techniques to implement to mosteffectively (e.g., most accurately, most computationally efficiently)monitor the writing device 106.

As described above, the augmented reality composition system 110 mayimplement a machine learning model to determine which techniques aremost effective in given circumstances. Alternatively, the augmentedreality composition system 110 may access user settings to determinewhich techniques to implement in given circumstances. For instance, auser 112 may set preferences to only use the emitter when the writingdevice 106 is moving slowly, or else to always track the writing device106 via the reflective elements, or else to always use the opticalsensor together with the IMU, but never to use object recognition (e.g.,to save battery). In any case, the augmented reality composition system110 may combine two or more of the techniques described with referenceto FIG. 5.

As also described above, the augmented reality composition system 110may change between techniques in real time as the user 112 draws withthe writing device 106. For example, in cases where the user 112 iscomposing outside, the ambient lighting may change with cloud cover astime passes, and the user may draw slowly at times and quickly at othertimes. Accordingly, the augmented reality composition system 110 detectssuch changes in the circumstances and changes between trackingtechniques accordingly in an on-the-fly manner in real time or near realtime.

To combine two or more techniques at a time, the augmented realitycomposition system 110 may take average values across the appliedtechniques. To illustrate, the augmented reality composition system 110may detect a tilt of the writing device 106 by way of the IMU and alsovia object recognition. The augmented reality composition system 110 maydetect a difference between the tilt determined via the IMU and the tiltdetermined via object recognition. In these cases, the augmented realitycomposition system 110 may average the two values to calculate the tiltof the writing device 106. In some embodiments, the augmented realitycomposition system 110 may apply different weights to the differenttechniques applied in a given circumstance depending on which techniquesare determined to be more effective in the given circumstance. Forexample, as the user 112 moves the writing device 106 slowly, theaugmented reality composition system 110 may determine, based on themachine learning model described above, that the optical sensor is moreaccurate than the IMU given the circumstances. Accordingly, theaugmented reality composition system 110 may still apply both techniquesbased on user settings (or as determined by the augmented realitycomposition system 110, as described above), but may apply a heavierweight to the readings of the optical sensor than to readings from theIMU when calculating an average of the two techniques.

Turning now to FIGS. 6A-6B, additional detail will be provided regardingcomponents of a writing device in accordance with one or moreembodiments. For example, FIG. 6A illustrates an example writing device600. In particular, the writing device 600 includes a housing 601 thatis composed of any suitable material such as, for example, metal,plastic, or a combination thereof. For example, the housing 601 mayinclude a pocket clip made of metal while the body of the housing 601 ismade of plastic. As another example, the housing 601 is composed of acontinuous piece of metal from end to end. In any case, the housing 601is made of material that is structurally sufficient for the user 112 tohold, squeeze, press, tap, and otherwise apply strain and pressure whilewriting without breaking.

Generally speaking, the housing 601 includes those portions of thewriting device 600 that are structural in nature—i.e., that connectother components of the writing device 600 together. As illustrated inFIG. 6A, the housing includes a cylindrical portion that extends from acircular back end including an eraser 620 toward a tip end that includesa tip 616. Near the tip 616, the housing 601 in FIG. 6A narrows to apoint in a conical shape that connects on one end to the cylindricalbody portion and connects on the other end to the tip 616.

In addition, the housing 601 need not be cylindrical in shape. Forinstance, in some embodiments, the housing 601 is prismatic—e.g., thehousing 601 may be a triangular prism, a rectangular prism, a hexagonalprism, an octagonal prism, or other prismatic shape. To illustrate, theback end of the writing device 600 that includes the eraser 620 may havea profile of a triangle when observed straight on. Accordingly, thehousing 601 extends from the back end of the writing device 600, whereeach side of the triangle distally extends the length of the writingdevice 600 to form a triangular prism to the portion near the tip of thewriting device 600 that narrows to the tip 616. Indeed, in theseembodiments, the narrowing portion is a triangular pyramid in shapewhere the base of the triangular pyramid connects with the triangularprism of the main portion of the housing 601, and where the top of thetriangular pyramid contains the tip 616.

In other embodiments, the eraser 620 is square (or rectangular) inprofile when observed straight on. In these embodiments, the housing 601extends from the eraser 620 along each side of the square to form arectangular prism of the body of the writing device 600. Likewise, inone or more embodiments, the housing 601 extends from the sides of theeraser 620 toward the tip 616.

Regardless of the shape of the housing 601, the housing 601 connectsand/or contains other components of the writing device 600. For example,the writing device 600 includes components in the form of chips orintegrated circuits that are configured for various purposes. Toillustrate, the writing device 600 includes an IMU 602, a pressuresensor chip 604, a BLUETOOTH chip 606, and an optics chip 614.Additional detail regarding the internal components of the writingdevice 600 is provided hereafter, after which detail regardingcorresponding external components of the writing device 600 is alsoprovided.

As mentioned, the writing device 600 includes an IMU 602. In particular,the IMU 602 includes one or more accelerometers, one or more gyroscopes,and/or one or more magnetometers. The IMU 602 gathers informationrelating to movement, tilt, and orientation of the writing device 600.To illustrate, the IMU 602 is configured to detect relative changes inposition as well as speed of movement in a lateral direction or avertical direction. The IMU 602 is further configured to detect changesin orientation is the writing device 600 spins or rotates, and isfurther configured to detect a rate of such changes (e.g., angularvelocity). The IMU 602 is also configured to detect changes in tilt ofthe writing device 600 by detecting an angle at which the writing device600 is leaning one direction or another. Accordingly, by communicatingwith the writing device 600 to collect information gathered by the IMU602, the augmented reality composition system 110 monitors changes inposition, tilt, and orientation of the writing device 600.

As also mentioned, the writing device 600 includes a pressure sensorchip 604. In particular, the pressure sensor chip 604 is configured tocollect and process information relating to pressure applied to one ormore pressure sensors of the writing device 600. For instance, thepressure sensor chip 604 gathers information indicating an amount ofpressure applied to the pressure sensor 618 by way of the tip 616 of thewriting device, and the pressure sensor chip 604 further gathersinformation indicating an amount of pressure and a timing of pressureapplied to the pressure sensor 622 by way of the eraser 620 of thewriting device 600. Thus, by communicating with the pressure sensor chip604, the augmented reality composition system 110 determines pressure(and varying degrees or amounts of pressure) applied to the tip 616 ofthe writing device and/or the eraser 620 of the writing device.

The writing device 600 further includes a BLUETOOTH chip 606 configuredto include or communicate with a BLUETOOTH antenna, and furtherconfigured to facilitate communications between the writing device 600and an augmented reality device (e.g., augmented reality device 108). Asdescribed above, the augmented reality composition system 110communicates with the writing device 600 to gather information from oneor more components of the writing device 600. For example, the augmentedreality composition system 110 gathers information from the IMU 602, thepressure sensor chip 604, and the optics chip 614. To facilitate suchcommunication, the writing device 600 includes a BLUETOOTH chip 606 oran alternative communication device operable to transmit sensoryinformation from the writing device 600 to an augmented reality device(e.g., augmented reality device 108) and/or to receive commands or othercommunications from the augmented reality device 108.

The writing device still further includes an optics chip 614 configuredto process sensory information captured by way of an optical sensor(e.g., optical sensor 612). In particular, the optics chip 614 includesone or more processors and is configured to process changes in light toconvert such changes into electrical signals indicating movement of thewriting device 600. Indeed, in some embodiments the optical sensor 612captures images one after another in succession as the writing device600 moves along a writing surface. The optics chip 614 collects andprocesses the images to detect differences within the images, and fromthose differences, determines a direction and speed of the writingdevice 600. In other embodiments, the optical sensor 612 projects alight and captures a portion of the projected light that reflects off ofthe writing surface back to the optical sensor 612. In theseembodiments, the optics chip 614 is configured to process the changes inreflected light to determine a direction and speed of movement of thewriting device 600. In any case, the augmented reality compositionsystem 110 communicates with the optics chip 614 to track movement ofthe writing device 600 by way of the optical sensor 612.

In addition to internal components such as integrated circuitcomponents, the writing device 600 also includes external and/ornon-circuit-based components such as reflective elements 608 a and 608b, emitter 610, optical sensor 612, and pressure sensors 618 and 622.The reflective elements 608 a and 608 b reflect a specific wavelength oflight such as, for example, infrared light. In these embodiments, theaugmented reality device 108 includes an infrared camera (or othersensor configured to detect light of the specific wavelength reflectedby the reflective elements 608 a and 608 b). By thus detecting lightreflected off of the reflective elements 608 a and 608 b, the augmentedreality composition system 110 may track movement of the writing device600 in three dimensions, and may further calculate a tilt of the writingdevice 600 by determining a position of reflective element 608 arelative to reflective element 608 b and calculating a slope betweenthem.

The reflective elements 608 a and 608 b may be separate from, andattached around, the housing 601. For example, in some embodiments thereflective elements 608 a and 608 b are infrared tape (e.g., tape thatcircumvents the housing 601). In other embodiments however, thereflective elements 608 a and 608 b are integrated within (e.g., a partof) the housing 601 and may be composed of the same material as thehousing 601, but further include reflective dye or some other coloringelement to reflect a specific range of wavelengths of light.Alternatively, the reflective elements 608 a and 608 b may each be acolored (e.g., painted) portion of the housing 601 where the coloringreflects the range of wavelengths of light.

As illustrated in FIG. 6A, the writing device 600 further includes anemitter 610. As described above, the emitter 610 is configured toproject a visual indicator onto a real-world surface. As shown in FIG.6A, the emitter 610 is affixed to the housing 601 at a location abovethe reflective element 608 b. In some embodiments, however, the emitter610 is affixed to the housing 601 at a location nearer to the tip 616 ofthe writing device (e.g., on the portion of the housing 601 that narrowsto the tip 616). In other embodiments, the emitter 610 is affixed to thehousing 601 at a location higher up on the housing 601 (e.g., halfwaybetween the eraser 620 and the tip 616, adjacent to the reflect element608 a, etc.). In some embodiments, the emitter 610 is not affixed to thehousing 601 but is instead integrated with (e.g., a part of) the housing601 (e.g., a continuous piece of metal, plastic, or other material).

Continuing with FIG. 6A, the writing device 600 also includes an opticalsensor 612. As mentioned, the optical sensor 612 is configured tocapture images or reflected light to detect changes within theimages/light such that the optics chip 614 can determine relativechanges in position and rates of positional change from the changes inthe images/light. While FIG. 6A depicts the optical sensor 612 locatedon the conical portion of the housing 601 that narrows to the tip 616,in some embodiments the optical sensor 612 is located within the tip616, or else is located higher up on the housing 601. In any case,similar to the emitter 610, the optical sensor 612 may be affixed to thehousing or else may be a part of the housing as a continuous piece ofmaterial.

As mentioned above, the writing device 600 further includes a tip 616and a pressure sensor 618. The tip 616 is configured to depress into theend of the writing device 600 and contact the pressure sensor 618 inresponse to forces applied to the writing device 600 by the user 112.While in some embodiments the tip 616 may move freely (e.g., withouttension or resistance) within the end of writing device 600, in otherembodiments the tip 616 may include a mechanism (e.g., a spring, apressure chamber, or simply a frictional fit within the end of thewriting device 600) configured to apply tension or resistance to the tip616. By adding resistance, the writing device 600 may be more tactilelyaccurate in replicating the sensation of writing or drawing with atraditional pen or pencil. Indeed, in some embodiments, the writingdevice 600 may adjust resistance of the tip 616 based on one or moresettings of the augmented reality composition system 110—e.g., when auser 112 selects to write with a pen setting, the resistance may bedifferent than when the user 112 selects to paint with a paintbrushsetting.

While FIG. 6A illustrates one embodiment of the pressure sensor 618,this is merely exemplary. It will be appreciated that the pressuresensor 618 may be of any suitable shape or size, and it will be furtherappreciated that the pressure sensor 618 may be located nearer to orfarther from the tip end of the writing device 600. Indeed, in someembodiments, the pressure sensor 618 may be an inductive sensor thatcircumscribes the tip 616 within the housing 601 and that detectschanges in induction as the tip (e.g., a metal tip) passes through thecircular pressure sensor 618 at varying depths or distances.

Furthermore, the writing device 600 includes an eraser 620 and apressure sensor 622. Similar to the discussion of the tip 616 and thepressure sensor 618, the eraser 620 is configured to depress into thewriting device 600 and contact the pressure sensor 622 with varyingdegrees of pressure. Additionally, although FIG. 6A illustrates aparticular shape and configuration of the eraser 620 and the pressuresensor 622, in some embodiments the eraser 620 and the pressure sensor622 may have different shapes and/or configurations. For example, theeraser 620 may be cylindrical without a protruding end shown in FIG. 6Asuch that the end of the eraser that is encased within the housing 601contacts the pressure sensor 622 as the user 112 presses down to erase adigital mark or augmented reality object.

While FIG. 6A illustrates one example embodiment of the writing device600, the writing device 600 may include additional or alternativeconfigurations. For example, the writing device 600 may have a housing601 of a different shape, as described above. Furthermore, the writingdevice 600 may include different configurations of the componentsdescribed above. Indeed, FIG. 6B describes an alternative embodiment ofa writing device 623 including various components.

As illustrated in FIG. 6B, the writing device 623 includes internalcircuitry components similar to those described above with referent toFIG. 6A. For example, the writing device 623 includes therein an IMU624, a pressure sensor chip 626, a BLUETOOTH chip 628, and an opticschip 630. Each of the IMU 624, the pressure sensor chip 626, theBLUETOOTH chip 628, and the optics chip 630 are configured as describedabove, and each is implemented by integrated circuits or othercircuitry-based components within the writing device 623.

As further illustrated by FIG. 6B, the writing device 623 includes ahousing 625. The housing 625 may include one or more componentsembedded. For example, the housing 625 may include an emitter 632configured to project a visual indicator onto a writing surface. Asdiscussed, by tracking the location of the visual indicator projected bythe emitter 632, the augmented reality composition system 110 monitorsthe position and/or movement of the writing device 623. In contrast toan emitter that protrudes out from the housing 625, the emitter 632 isflush with the housing 625 or else inset within the housing 625. WhileFIG. 6B illustrates the emitter 632 located in one particular locationwithin the housing 625, in other embodiments the emitter 632 may belocated in alternative locations, such as nearer to the tip 636 of thewriting device 623.

FIG. 6B further illustrates a reflective element 634. As describedabove, the reflective element 634 is configured to reflect a range oflight wavelengths. In contrast to FIG. 6A, the writing device 623 ofFIG. 6B includes a single reflective element 634 located near the centerof the housing 625 of the writing device 623. While FIG. 6B illustratesa particular size of the reflective element 634, in some embodiments thereflective element 634 may be larger or smaller. Additionally, thereflective element 634 may be located in alternative locations on thehousing 625 (or as part of the housing 625, as described in relation toFIG. 6A).

The writing device 623 further includes a tip 636 configured to contacta pressure sensor 638, as described above. Additionally, the tip 636 mayinclude an optical sensor 640 located therein. The optical sensor 640 isconfigured to detect changes in light or images, as described above. Byincluding the optical sensor 640 within the tip 636, the writing device623 may more accurately track changes in position because the opticaldevice 640 is closer to a given writing surface.

As further illustrated in FIG. 6B, the writing device 623 includes aneraser 642 and a pressure sensor 644. The eraser 642 and the pressuresensor 644 are configured as similarly described above in relation toFIG. 6A. For example, the eraser 642 is configured to depress into theback end of the housing 625 to contact the pressure sensor 644 withvarying degrees of pressure. Additionally, the eraser 642 may bedifferent shape than the eraser 642 illustrated in FIG. 6B. For example,the eraser 642 may be entirely cylindrical and may not include a nub, asdescribed above.

Although FIGS. 6A-6B illustrate example embodiments of writing devices600 and 623, respectively, it will be understood from the disclosureherein that, in some embodiments, writing device 600 includes more orfewer components than those depicted in FIG. 6A, and likewise, writingdevice 623 includes more or fewer components than those depicted in FIG.6B. In addition, the writing devices 600 and 623 may also includealternative or additional arrangements of the components depicted inFIGS. 6A and 6B, respectively. For example, as described above, thewriting device 623 of FIG. 6B may include a reflective element 634 thatdoes not entirely circumscribe the housing 625 but instead includes agap, or else includes a specific pattern, at one or more locations. Byincluding gaps or patterns at known locations, the augmented realitycomposition system 110 may determine an orientation of the writingdevice 623 via the reflective element 634 based on portions of thereflective element 634 that are visible to the augmented reality device108.

Although FIGS. 6A-6B illustrate example embodiments of writing devices,a writing device may include additional or alternative arrangements ofcomponents to those illustrated in FIGS. 6A-6B. For example, in someembodiments a writing device includes an emitter 610 and an opticalsensor 612, but does not include reflective elements 608. In otherembodiments, the writing device includes reflective elements 608 and theoptical sensor 612, but does not include the emitter 610. In one or moreembodiments, the writing device includes multiple emitters. In any case,the writing device 106 described in the various embodiments 106 a-106 dof FIGS. 6A-6B may include any combination of the components andfeatures described herein.

Looking now to FIG. 7, additional detail will be provided regardingcomponents and capabilities of the augmented reality composition system110. Specifically, FIG. 7 illustrates an example schematic diagram ofthe augmented reality composition system 110 on an example computingdevice 700 (e.g., the augmented reality device 108, the writing device106, and/or the server(s) 102). As shown in FIG. 7, the augmentedreality composition system 110 may include an environment analyzer 702,a sensor information manager 704, an object recognizer 706, a movementdetector 708, an augmented reality rendering manager 710, and a storagemanager 712. While FIG. 7 depicts a particular number of components, insome embodiments, the augmented reality composition system 110 mayinclude more or fewer components. In addition, the components mayperform additional or alternative tasks than those described hereafter.

As illustrated in FIG. 7, the augmented reality composition system 110includes an environment analyzer 702. In particular, the environmentanalyzer 702 examines, scans, evaluates, recognizes, parses, orotherwise analyzes a real-world environment. For example, theenvironment analyzer 702 analyzes a view of a real-world environmentthat is visible through an augmented reality device 108 to identifyreal-world surfaces within the real world that are acceptable and/orunacceptable for composing, as described above. Additionally, theenvironment analyzer 702 communicates with the augmented realityrendering manager 710 to render indicators, overlays, etc., to designatethe real-world surfaces as acceptable or not within an augmented realityenvironment presented to the user 112 by way of the augmented realitydevice 108.

In addition, the augmented reality composition system 110 of FIG. 7includes a sensor information manager 704. In particular, the sensorinformation manager 704 collects, gathers, transmits, reads, analyzes,extrapolates, and/or interprets sensory information. For example, thesensor information manager 704 collects information by way of a writingdevice 106 pertaining to a pressure sensor input. The sensor informationmanager 704 further gathers sensory information by way of an augmentedreality device 108 in response to detecting one or more reflectiveelements (e.g., reflective elements 608) on the writing device 106 byway of a camera or other sensor. Additionally, the sensor informationmanager 704 receives information from the writing device 106corresponding to IMU data associated with a tilt and/or movement of thewriting device 106. The sensor information manager 704 also gathersinformation by way of an augmented reality device 108 by recognizing alocation of a visual indicator projected by an emitter (e.g., emitter610) associated with the writing device 106. The sensor manager 704 mayalso or alternatively collect sensory information from the writingdevice relating to the optical sensor (e.g., optical sensor 612) as thewriting device 106 moves on a surface.

The sensor information manager 704 communicates with the movementdetector 708 to determine whether the location of the writing device 106is constant or changing. For instance, in response to collecting sensoryinformation pertaining to a location of a visual indicator at a firstlocation and subsequently collecting sensory information pertaining to asecond location of the visual indicator, the movement detector 708 maydetermine that the writing device 106 is moving.

As further illustrated in FIG. 7, the augmented reality compositionsystem 110 may include an object recognizer 706. In particular, theobject recognizer 706 may communicate with the sensor informationmanager 704 relating to information of reflective elements, an opticalsensor, an emitter, etc., to recognize the writing device 106. In someembodiments, the object recognizer 706 may analyze the view of theaugmented reality device 108 and perform object recognition techniquesas described herein to identify the writing device 106. Similarly, theobject recognizer may perform spatial recognition algorithms to identifyreal-world surfaces and to further determine which of the real-worldsurfaces are acceptable for composing and which are not.

As illustrated in FIG. 7, the augmented reality composition system 110may also include a movement detector 708. The movement detector 708 maycommunicate with the object recognizer 706 and/or the sensor informationmanager 704 to detect movement of the writing device 106. In addition,the movement detector 708 can detect an initial position of the writingdevice 106 in accordance with the methods described herein.

As further illustrated in FIG. 7, the augmented reality compositionsystem 110 may include an augmented reality rendering manager 710. Inparticular, the augmented reality rendering manager 710 may communicatewith the object recognizer 706 and the movement detector 708 to generatean augmented reality environment as described herein. Additionally, theaugmented reality rendering manager 710 may generate augmented realityobjects within the augmented reality environment for presentation to theuser 112. The augmented reality objects correspond to movement detectedby the movement detector 708, as described in further detail above.

As mentioned, the augmented reality composition system 110 of FIG. 7further includes a storage manager 712. The storage manager 712 mayinclude sensory data 714 and environment data 716. In particular, thestorage manager 712 may communicate with the sensor information manager704 to collect and store sensory data 714 pertaining to the componentsand features of the writing device described herein. Additionally, thestorage manager 712 may communicate with the environment analyzer 702 tostore information (e.g., three-dimensional coordinates, size, etc.)pertaining to the real-world environment that the augmented realitycomposition system 110 analyzes. By storing the environment data 716,the augmented reality composition system 110 can remember previoussurfaces, augmented reality objects, etc., for areas that are notcurrently within the view of the augmented reality device 108, asdiscussed above.

FIGS. 1-7, the corresponding text, and the examples provide a number ofdifferent systems and methods that generate augmented reality objectsbased on tracking movement of a writing device. In addition to theforegoing, embodiments can also be described in terms of flowchartscomprising acts in a method for accomplishing a particular result. Forexample, FIG. 8 illustrates a flowchart of an exemplary method inaccordance with one or more embodiments.

While FIG. 8 illustrates acts according to one embodiment, alternativeembodiments may omit, add to, reorder, and/or modify any of the actsshown in FIG. 8. The acts of FIG. 8 can be performed as part of amethod. Alternatively, a non-transitory computer readable medium cancomprise instructions, that when executed by one or more processors,cause a computing device to perform the acts of FIG. 8. In still furtherembodiments, a system can perform the acts of FIG. 8. Additionally, thesteps/acts described herein may be repeated or performed in parallelwith one another or in parallel with different instances of the same orother similar steps/acts.

FIG. 8 illustrates an exemplary method (or series of acts) 800 ofrendering an augmented reality object based on tracking movement of awriting device. The method 800 includes an act 802 of detecting aninitial contact of the writing device. In particular, the act 802 caninvolve detecting, by way of a pressure sensor of a writing device, aninitial contact of the writing device with a real-world surface.

The method 800 of FIG. 8 also includes an act 804 of determining aninitial location of the writing device. In particular, the act 804 caninvolve determining, in response to the detected initial contact and byway of a locator element associated with the writing device, an initiallocation of the writing device. In some embodiments, the locator elementcan include one or more of: a reflective element operable to reflect arange of light wavelengths measurable by the augmented reality device,an emitter operable to project a visual indicator onto the real-worldsurface, or a shape of the writing device recognizable by the augmentedreality device.

In addition, the method 800 includes an act 806 of detecting real-worldmovement of the writing device. In particular, the act 806 can involvedetecting, by way of a motion detector element associated with thewriting device, real-world movement of the writing device relative tothe initial location. In some embodiments, the motion detector elementcan include one or more of: a reflective element operable to reflect arange of light wavelengths measurable by the augmented reality device,an emitter operable to project a visual indicator onto the real-worldsurface, a shape of the writing device recognizable by the augmentedreality device, an optical sensor operable to detect light deflection ofa light emitted by the writing device off of the real-world surface, oran inertial measurement unit.

As further illustrated by FIG. 8, the method 800 includes an act 808 ofrendering an augmented reality object. In particular, the act 808 caninclude rendering, by way of an augmented reality device and based onthe detected real-world movement of the writing device, an augmentedreality object such that the augmented reality object appears to bedrawn on the real-world surface. Rendering the augmented reality objectcan include detecting, by way of the pressure sensor associated with thewriting device, that the writing device is contacting the real-worldsurface, and generating, by way of the augmented reality device and inresponse to detecting that the writing device is contacting thereal-world surface, a digital mark that appears to be drawn on thereal-world surface and that follows the detected real-world movement ofthe writing device.

Though not illustrated in FIG. 8, the method 800 may further include anact of analyzing, by way of the augmented reality device, a real-worldenvironment to identify at least one real-world surface acceptable forcomposing. In addition, the method 800 may include an act of providing,by way of the augmented reality device and in response to analyzing theat least one real-world surface, an indication that the at least onereal-world surface is acceptable for composing. Furthermore, the method800 can include acts of determining that the real-world movement of thewriting device is above a speed threshold, determining, in response todetermining that the real-world movement is above the speed threshold,the real-world movement of the writing device based on a first motiondetector element, and, in response to determining that the real-worldmovement is no longer above the speed threshold, determining thereal-world movement of the writing device based on a second motiondetector element.

Furthermore, the method 800 can include an act of detecting, by way ofthe pressure sensor associated with the writing device, a pressure forceapplied to the writing device, as well as an act of detecting, by way ofan inertial measurement unit associated with the writing device, a tiltof the writing device. The method 800 may also include an act ofadjusting a weight of the digital mark generated by way of the augmentedreality device based on the detected pressure force applied to thewriting device and the detected tilt of the writing device. Furthermore,the method 800 may include an act of generating, based on the renderedaugmented reality object, a three-dimensional augmented reality modelcorresponding to the augmented reality object, wherein thethree-dimensional augmented reality model is manipulable by a userwithin a three-dimensional augmented reality environment.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 9 illustrates, in block diagram form, an exemplary computing device900 that may be configured to perform one or more of the processesdescribed above. One will appreciate that the augmented realitycomposition system 110 can comprise implementations of the computingdevice 900. As shown by FIG. 9, the computing device can comprise aprocessor 902, memory 904, a storage device 906, an I/O interface 908,and a communication interface 910. In certain embodiments, the computingdevice 900 can include fewer or more components than those shown in FIG.9. Components of computing device 900 shown in FIG. 9 will now bedescribed in additional detail.

In particular embodiments, processor(s) 902 includes hardware forexecuting instructions, such as those making up a computer program. Asan example, and not by way of limitation, to execute instructions,processor(s) 902 may retrieve (or fetch) the instructions from aninternal register, an internal cache, memory 904, or a storage device906 and decode and execute them.

The computing device 900 includes memory 904, which is coupled to theprocessor(s) 902. The memory 904 may be used for storing data, metadata,and programs for execution by the processor(s). The memory 904 mayinclude one or more of volatile and non-volatile memories, such asRandom Access Memory (“RAM”), Read Only Memory (“ROM”), a solid statedisk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of datastorage. The memory 904 may be internal or distributed memory.

The computing device 900 includes a storage device 906 includes storagefor storing data or instructions. As an example, and not by way oflimitation, storage device 906 can comprise a non-transitory storagemedium described above. The storage device 906 may include a hard diskdrive (HDD), flash memory, a Universal Serial Bus (USB) drive or acombination of these or other storage devices.

The computing device 900 also includes one or more input or output(“I/O”) devices/interfaces 908, which are provided to allow a user toprovide input to (such as user strokes), receive output from, andotherwise transfer data to and from the computing device 900. These I/Odevices/interfaces 908 may include a mouse, keypad or a keyboard, atouch screen, camera, optical scanner, network interface, modem, otherknown I/O devices or a combination of such I/O devices/interfaces 908.The touch screen may be activated with a writing device or a finger.

The I/O devices/interfaces 908 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, devices/interfaces 908 isconfigured to provide graphical data to a display for presentation to auser. The graphical data may be representative of one or more graphicaluser interfaces and/or any other graphical content as may serve aparticular implementation.

The computing device 900 can further include a communication interface910. The communication interface 910 can include hardware, software, orboth. The communication interface 910 can provide one or more interfacesfor communication (such as, for example, packet-based communication)between the computing device and one or more other computing devices 900or one or more networks. As an example, and not by way of limitation,communication interface 910 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI. The computingdevice 900 can further include a bus 911. The bus 911 can comprisehardware, software, or both that couples components of computing device900 to each other.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with less or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel with one another or inparallel with different instances of the same or similar steps/acts. Thescope of the invention is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes that come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

1. In a digital medium environment for generating augmented realityobjects, a computer-implemented method of creating and arranging adigital object on an augmented reality representation of a real-worlddrawing space comprising: detecting, by way of a pressure sensor of awriting device, an initial contact of the writing device with areal-world surface; determining, in response to the detected initialcontact and by way of a locator element associated with the writingdevice, an initial location of the writing device; detecting real-worldmovement of a tip of the writing device relative to the initial locationby: tracking motion of a reflective motion detector element located onthe writing device and that is operable to reflect light within a rangeof light wavelengths; and determining, during the real-world movement, aposition of the tip of the writing device relative to the reflectivemotion detector element; and rendering, by way of an augmented realitydevice and based on the detected real-world movement of the tip of thewriting device, an augmented reality object such that the augmentedreality object appears to be drawn on the real-world surface.
 2. Thecomputer-implemented method of claim 1, further comprising analyzing, byway of the augmented reality device, a real-world environment toidentify at least one real-world surface acceptable for composing. 3.The computer-implemented method of claim 2, further comprisingproviding, by way of the augmented reality device and in response toanalyzing the at least one real-world surface, an indication that the atleast one real-world surface is acceptable for composing.
 4. Thecomputer-implemented method of claim 1, further comprising: determiningthat the real-world movement of the writing device is above a speedthreshold; in response to determining that the real-world movement isabove the speed threshold, determining the real-world movement of thewriting device based on a first motion detector element; and in responseto determining that the real-world movement is no longer above the speedthreshold, determining the real-world movement of the writing devicebased on a second motion detector element.
 5. The computer-implementedmethod of claim 1, further comprising: detecting an environmentalcircumstance associated with the writing device that affects trackingthe writing device; and determining, based on the detected environmentalcircumstance, one or more techniques to utilize to monitor the writingdevice.
 6. The computer-implemented method of claim 5, wherein:detecting the environmental circumstance comprises detecting that thewriting device is writing on a non-reflective surface; and determiningthe one or more techniques to utilize to monitor the writing devicecomprises selecting, based on detecting that the writing device iswriting on the non-reflective surface, one or more of the reflectivemotion detector element, an emitter operable to project a visualindicator onto the non-reflective surface, an optical sensor operable tocapture images corresponding to positional changes of the writingdevice, or a machine learning model.
 7. The computer-implemented methodof claim 1, wherein rendering the augmented reality object comprises:detecting, by way of the pressure sensor associated with the writingdevice, that the writing device is contacting the real-world surface;and generating, by way of the augmented reality device and in responseto detecting that the writing device is contacting the real-worldsurface, a digital mark that appears to be drawn on the real-worldsurface and that follows the detected real-world movement of the writingdevice.
 8. The computer-implemented method of claim 7, furthercomprising: detecting, by way of the pressure sensor associated with thewriting device, a pressure force applied to the writing device;detecting, by way of an inertial measurement unit associated with thewriting device, a tilt of the writing device; and adjusting acompositional attribute of the digital mark generated by way of theaugmented reality device based on the detected pressure force applied tothe writing device and the detected tilt of the writing device.
 9. Thecomputer-implemented method of claim 1, further comprising generating,based on the rendered augmented reality object, a three-dimensionalaugmented reality model corresponding to the augmented reality object,wherein the three-dimensional augmented reality model is manipulable bya user within a three-dimensional augmented reality environment.
 10. Anon-transitory computer readable storage medium comprising instructionsthereon that, when executed by at least one processor, cause a computersystem to: detect, by way of a pressure sensor of a writing device, aninitial contact of the writing device with a real-world surface;determine, in response to the detected initial contact and by way of alocator element associated with the writing device, an initial locationof the writing device; detect real-world movement of a tip of thewriting device relative to the initial location by: tracking motion of amotion detector element located on the writing device, wherein themotion detector element comprises at least one of a reflective elementoperable to reflect a light within a range of light wavelengths or anemitter operable to project a trackable visual indicator onto thereal-world surface; and determining, during the real-world movement, aposition of the tip of the writing device relative to the motiondetector element; and render, by way of an augmented reality device andbased on the detected real-world movement of the tip of the writingdevice, an augmented reality object such that the augmented realityobject appears to be drawn on the real-world surface.
 11. Thenon-transitory computer readable storage medium of claim 10, furthercomprising instructions thereon that, when executed by the at least oneprocessor, cause the computer system to analyze, by way of the augmentedreality device, a real-world environment to identify at least onereal-world surface acceptable for composing.
 12. The non-transitorycomputer readable storage medium of claim 11, further comprisinginstructions thereon that, when executed by the at least one processor,cause the computer system to provide, by way of the augmented realitydevice and in response to analyzing the at least one real-world surface,an indication that the at least one real-world surface is acceptable forcomposing.
 13. The non-transitory computer readable storage medium ofclaim 10, further comprising instructions thereon that, when executed bythe at least one processor, cause the computer system to: determine thatthe real-world movement of the writing device is above a speedthreshold; in response to determining that the real-world movement isabove the speed threshold, determine the real-world movement of thewriting based on a first motion detector; and in response to determiningthat the real-world movement is no longer above the speed threshold,determine the real-world movement of the writing device based on asecond motion detector.
 14. The non-transitory computer readable storagemedium of claim 10, wherein the locator element comprises one or moreof: a reflective element operable to reflect a range of lightwavelengths measurable by the augmented reality device or an emitteroperable to project a visual indicator onto the real-world surface. 15.The non-transitory computer readable storage medium of claim 10, whereinthe motion detector element is the reflective element and furthercomprising instructions that, when executed by the at least oneprocessor, cause the computer system to detect the real-world movementof the writing device by tracking motion of the reflective element usinga sensor operable to detect light within the range of light wavelengths.16. The non-transitory computer readable storage medium of claim 10,wherein detecting the real-world movement of the writing devicecomprises training a machine learning model to predict, based onanalyzing writing motions of a plurality of individuals, where a userwill move the writing device.
 17. A writing device for composingaugmented reality objects for display by way of an augmented realitydevice based on tracking movement of the writing device on a real-worldsurface, the writing device comprising: a housing having a tip on afirst end of the housing; a pressure sensor within the housing that,when triggered, indicates a pressure force applied to the tip; aninertial measurement unit within the housing, comprising at least oneaccelerometer, that detects at least one of: a location of the writingdevice or a tilt of the writing device; a locator element operable toindicate a location of the writing device, the locator element beingaffixed to the housing and comprising at least one of: an emitter thatprojects a visual indicator onto a writing surface or a reflectiveelement that reflects a range of light wavelengths measurable by anaugmented reality device; a reflective motion detector element operableto indicate real-world movement the a tip of the housing based on aposition of the tip of the housing relative to the reflective motiondetector element; and a transceiver element affixed to the housing thatsends information associated with at least one of: the pressure sensor,the inertial measurement unit, the locator element, or the reflectivemotion detector element.
 18. The writing device of claim 17, wherein thelocator element comprises the reflective element and the reflectiveelement comprises an infrared element that reflects infrared lightwavelengths.
 19. The writing device of claim 17, wherein the locatorelement comprises the emitter and the emitter comprises a laser emitter.20. The writing device of claim 17, further comprising an optical sensorthat tracks movement of the writing device based on light deflectionfrom the writing surface.